Open-Source Pre-Processing Tools for Unstructured Data
+面向非结构化数据的开源预处理工具
-The `unstructured` library provides open-source components for ingesting and pre-processing images and text documents, such as PDFs, HTML, Word docs, and [many more](https://docs.unstructured.io/open-source/core-functionality/partitioning). The use cases of `unstructured` revolve around streamlining and optimizing the data processing workflow for LLMs. `unstructured` modular functions and connectors form a cohesive system that simplifies data ingestion and pre-processing, making it adaptable to different platforms and efficient in transforming unstructured data into structured outputs. +`unstructured` 库提供用于接入与预处理图像和文本文档的开源组件，例如 PDF、HTML、Word 文档，以及[更多格式](https://docs.unstructured.io/open-source/core-functionality/partitioning).。`unstructured` 的用例围绕简化并优化面向 LLM 的数据处理工作流。`unstructured` 的模块化函数与连接器构成一个内聚系统，简化数据接入与预处理，使其可适配不同平台，并能高效地将非结构化数据转换为结构化输出。 -## Unstructured Transform MCP — Document Processing for your Agents +## Unstructured Transform MCP — 为你的智能体提供文档处理能力 -Unstructured Transform brings production-grade document processing to your agents as an MCP server. It gives them the ability to turn 60+ file types into structured data that's ready for your applications, vector databases, and any downstream processes by parsing, enriching, chunking, and embedding files directly inside their current session. +Unstructured Transform 以 MCP 服务器的形式，将生产级文档处理能力带给智能体。它使智能体能够将 60 多种文件类型转换为结构化数据，可直接用于你的应用、向量数据库以及任意下游流程——通过在当前会话内直接完成文件的解析、增强、分块与嵌入。 -### Setup Steps for Your Agent +### 为智能体配置的步骤 -1. **Pick your MCP client.** Transform works with virtually any MCP-compatible host or agent framework — Claude Code, Cursor, Codex CLI and more. +1. **选择你的 MCP 客户端。** Transform 几乎可与任何兼容 MCP 的宿主或智能体框架配合使用——Claude Code、Cursor、Codex CLI 等。 -2. **Add the Transform MCP server** to your client's MCP configuration (via the CLI `mcp add` command or the client's MCP settings/config file, depending on the tool). +2. **将 Transform MCP 服务器添加**到客户端的 MCP 配置中（可通过 CLI 的 `mcp add` 命令，或客户端的 MCP 设置/配置文件，具体取决于所用工具）。 -3. **Authenticate once** when your client prompts you. Sign in, and the Transform tools become available to your agent on its next message. +3. **在客户端提示时完成一次身份验证。** 登录后，Transform 工具会在智能体的下一条消息中可用。 -4. **Point your agent at a file.** Drag and drop or reference a local file or URL. Transform handles 60+ formats (PDFs, emails, images, scanned files, and more). +4. **让智能体指向某个文件。** 拖放或引用本地文件或 URL。Transform 支持 60 多种格式（PDF、邮件、图像、扫描件等）。 -5. **Describe what you need in plain language.** Tell the agent your intent (e.g. "parse and chunk this contract for a vector store") and Transform partitions, enriches, chunks, and embeds the file, returning structured data ready to use. +5. **用自然语言描述你的需求。** 告诉智能体你的意图（例如「解析并分块这份合同，用于向量存储」），Transform 会完成文件的分区、增强、分块与嵌入，并返回可直接使用的结构化数据。 -15,000 free pages a month, 3 cents per page after! +每月 15,000 页免费，超出后每页 3 美分！ -📄 Full docs: https://docs.unstructured.io/transform/overview +📄 完整文档：https://docs.unstructured.io/transform/overview ## Unstructured Pipelines -Ready to move your data processing pipeline to production, and take advantage of advanced features? Check out [Unstructured Platform](https://unstructured.io/enterprise). In addition to better processing performance, take advantage of chunking, embedding, and image and table enrichment generation, all from a low code UI or an API. [Request a demo](https://unstructured.io/contact) from our sales team to learn more about how to get started. +准备将数据处理流水线推向生产，并使用高级功能？请了解 [Unstructured Platform](https://unstructured.io/enterprise).。除更好的处理性能外，还可使用分块、嵌入，以及图像与表格增强生成，全部可通过低代码 UI 或 API 完成。向我们的销售团队[申请演示](https://unstructured.io/contact)，了解如何开始使用。 -## :eight_pointed_black_star: Quick Start +## :eight_pointed_black_star: 快速开始 -There are several ways to use the `unstructured` library: -* [Run the library in a container](https://github.com/Unstructured-IO/unstructured#run-the-library-in-a-container) or -* Install the library - 1. [Install from PyPI](https://github.com/Unstructured-IO/unstructured#installing-the-library) - 2. [Install for local development](https://github.com/Unstructured-IO/unstructured#installation-instructions-for-local-development) -* For installation with `conda` on Windows system, please refer to the [documentation](https://unstructured-io.github.io/unstructured/installing.html#installation-with-conda-on-windows) +使用 `unstructured` 库有多种方式： +* [在容器中运行该库](https://github.com/Unstructured-IO/unstructured#run-the-library-in-a-container)，或 +* 安装该库 + 1. [从 PyPI 安装](https://github.com/Unstructured-IO/unstructured#installing-the-library) + 2. [本地开发安装](https://github.com/Unstructured-IO/unstructured#installation-instructions-for-local-development) +* 在 Windows 系统上使用 `conda` 安装时，请参阅[文档](https://unstructured-io.github.io/unstructured/installing.html#installation-with-conda-on-windows) -### Run the library in a container +### 在容器中运行该库 -The following instructions are intended to help you get up and running using Docker to interact with `unstructured`. -See [here](https://docs.docker.com/get-docker/) if you don't already have docker installed on your machine. +以下说明旨在帮助你使用 Docker 与 `unstructured` 交互并完成上手。 +若你的机器上尚未安装 Docker，请参阅[此处](https://docs.docker.com/get-docker/)。 -NOTE: we build multi-platform images to support both x86_64 and Apple silicon hardware. `docker pull` should download the corresponding image for your architecture, but you can specify with `--platform` (e.g. `--platform linux/amd64`) if needed. +注意：我们构建多平台镜像以同时支持 x86_64 与 Apple silicon 硬件。`docker pull` 应会为你的架构下载对应镜像，但如有需要，也可通过 `--platform` 指定（例如 `--platform linux/amd64`）。 -We build Docker images for all pushes to `main`. We tag each image with the corresponding short commit hash (e.g. `fbc7a69`) and the application version (e.g. `0.5.5-dev1`). We also tag the most recent image with `latest`. To leverage this, `docker pull` from our image repository. +我们会为推送到 `main` 的每次提交构建 Docker 镜像。每个镜像都会打上对应的短提交哈希标签（例如 `fbc7a69`）以及应用版本标签（例如 `0.5.5-dev1`）。我们还会将最新镜像标记为 `latest`。要利用这一点，请从我们的镜像仓库执行 `docker pull`。 ```bash docker pull downloads.unstructured.io/unstructured-io/unstructured:latest ``` -Once pulled, you can create a container from this image and shell to it. +拉取完成后，你可以基于该镜像创建容器并进入其 shell。 ```bash # create the container @@ -97,13 +103,9 @@ docker run -dt --name unstructured downloads.unstructured.io/unstructured-io/uns docker exec -it unstructured bash ``` -You can also build your own Docker image. Note that the base image is `wolfi-base`, which is -updated regularly. If you are building the image locally, it is possible `docker-build` could -fail due to upstream changes in `wolfi-base`. +你也可以自行构建 Docker 镜像。请注意，基础镜像是 `wolfi-base`，会定期更新。若在本地构建镜像，`docker-build` 可能因 `wolfi-base` 的上游变更而失败。 -If you only plan on parsing one type of data you can speed up building the image by commenting out some -of the packages/requirements necessary for other data types. See Dockerfile to know which lines are necessary -for your use case. +若你仅计划解析一种数据类型，可通过注释掉其他数据类型所需的若干包/依赖项来加快镜像构建。请查看 Dockerfile 以了解你的使用场景需要保留哪些行。 ```bash make docker-build @@ -112,7 +114,7 @@ make docker-build make docker-start-bash ``` -Once in the running container, you can try things directly in Python interpreter's interactive mode. +进入运行中的容器后，你可以直接在 Python 解释器的交互模式中尝试相关操作。 ```bash # this will drop you into a python console so you can run the below partition functions python3 @@ -124,25 +126,24 @@ python3 >>> elements = partition_text(filename="example-docs/fake-text.txt") ``` -### Installing the library -Use the following instructions to get up and running with `unstructured` and test your -installation. +### 安装该库 -- Install the Python SDK to support all document types with `pip install "unstructured[all-docs]"` - - For plain text files, HTML, XML, JSON and Emails that do not require any extra dependencies, you can run `pip install unstructured` - - To process other doc types, you can install the extras required for those documents, such as `pip install "unstructured[docx,pptx]"` -- Install the following system dependencies if they are not already available on your system. - Depending on what document types you're parsing, you may not need all of these. - - `libmagic-dev` (filetype detection) - - `poppler-utils` (images and PDFs) - - `tesseract-ocr` (images and PDFs, install `tesseract-lang` for additional language support) - - `libreoffice` (MS Office docs) - - `pandoc` is bundled automatically via the `pypandoc-binary` Python package (no system install needed) +请按以下说明安装 `unstructured` 并测试你的安装。 -- For suggestions on how to install on the Windows and to learn about dependencies for other features, see the - installation documentation [here](https://unstructured-io.github.io/unstructured/installing.html). +- 使用 `pip install "unstructured[all-docs]"` 安装 Python SDK，以支持所有文档类型 + - 对于纯文本文件、HTML、XML、JSON 以及无需额外依赖的 Email，你可以运行 `pip install unstructured` + - 要处理其他文档类型，可安装相应文档所需的 extras，例如 `pip install "unstructured[docx,pptx]"` +- 若你的系统上尚未提供以下系统依赖，请先安装。 + 具体需要哪些，取决于你要解析的文档类型，未必全部都需要。 + - `libmagic-dev`（文件类型检测） + - `poppler-utils`（图像与 PDF） + - `tesseract-ocr`（图像与 PDF；如需额外语言支持，请安装 `tesseract-lang`） + - `libreoffice`（MS Office 文档） + - `pandoc` 会通过 `pypandoc-binary` Python 包自动捆绑（无需系统级安装） -At this point, you should be able to run the following code: +- 关于在 Windows 上的安装建议，以及了解其他功能所需的依赖，请参阅[此处](https://unstructured-io.github.io/unstructured/installing.html). 的安装文档。 + +至此，你应能运行以下代码： ```python from unstructured.partition.auto import partition @@ -151,70 +152,63 @@ elements = partition(filename="example-docs/eml/fake-email.eml") print("\n\n".join([str(el) for el in elements])) ``` -### Installation Instructions for Local Development +### 本地开发安装说明 -The following instructions are intended to help you get up and running with `unstructured` -locally if you are planning to contribute to the project. +若你计划为该项目贡献代码，以下说明旨在帮助你在本地安装并运行 `unstructured`。 -This project uses [uv](https://docs.astral.sh/uv/) for dependency management. Install it first: +本项目使用 [uv](https://docs.astral.sh/uv/) 进行依赖管理。请先安装它： ```bash # macOS / Linux curl -LsSf https://astral.sh/uv/install.sh | sh ``` -Then install all dependencies (base, extras, dev, test, and lint groups): +然后安装所有依赖（base、extras、dev、test 与 lint 组）： ```bash make install ``` -This runs `uv sync --locked --all-extras --all-groups`, which creates a virtual environment -and installs everything in one step. No need to manually create or activate a virtualenv. +这会运行 `uv sync --locked --all-extras --all-groups`，它会创建虚拟环境并一步完成全部安装。无需手动创建或激活 virtualenv。 -To install only specific document-type extras: +若仅需安装特定文档类型的 extras： ```bash uv sync --extra pdf uv sync --extra csv --extra docx ``` -To update the lock file after changing dependencies in `pyproject.toml`: +若在 `pyproject.toml` 中修改了依赖，要更新锁文件，请执行： ```bash make lock ``` -* Optional: - * To install extras for processing images and PDFs locally, run `uv sync --extra pdf --extra image`. - * For processing image files, `tesseract` is required. See [here](https://tesseract-ocr.github.io/tessdoc/Installation.html) for installation instructions. - * For processing PDF files, `tesseract` and `poppler` are required. The [pdf2image docs](https://pdf2image.readthedocs.io/en/latest/installation.html) have instructions on installing `poppler` across various platforms. +* 可选： + * 若要在本地安装用于处理图像与 PDF 的 extras，请运行 `uv sync --extra pdf --extra image`。 + * 处理图像文件需要 `tesseract`。安装说明请参阅[此处](https://tesseract-ocr.github.io/tessdoc/Installation.html)。 + * 处理 PDF 文件需要 `tesseract` 与 `poppler`。[pdf2image docs](https://pdf2image.readthedocs.io/en/latest/installation.html) 提供了在各平台上安装 `poppler` 的说明。 -Additionally, if you're planning to contribute to `unstructured`, we provide you an optional `pre-commit` configuration -file to ensure your code matches the formatting and linting standards used in `unstructured`. -If you'd prefer not to have code changes auto-tidied before every commit, you can use `make check` to see -whether any linting or formatting changes should be applied, and `make tidy` to apply them. +此外，若你计划为 `unstructured` 贡献代码，我们提供了一个可选的 `pre-commit` 配置文件，以确保你的代码符合 `unstructured` 中使用的格式化与 lint 标准。 +若你不希望在每次提交前自动整理代码变更，可使用 `make check` 查看是否应应用任何 lint 或格式化变更，并使用 `make tidy` 来应用它们。 -If using the optional `pre-commit`, you'll just need to install the hooks with `pre-commit install` since the -`pre-commit` package is installed as part of `make install` mentioned above. Finally, if you decided to use `pre-commit` -you can also uninstall the hooks with `pre-commit uninstall`. +若使用可选的 `pre-commit`，你只需使用 `pre-commit install` 安装 hooks，因为上文提到的 `make install` 已包含 `pre-commit` 包的安装。最后，若你决定使用 `pre-commit`，也可使用 `pre-commit uninstall` 卸载 hooks。 -In addition to develop in your local OS we also provide a helper to use docker providing a development environment: +除在本地操作系统上开发外，我们还提供了一个借助 Docker 搭建开发环境的辅助工具： ```bash make docker-start-dev ``` -This starts a docker container with your local repo mounted to `/mnt/local_unstructured`. This docker image allows you to develop without worrying about your OS's compatibility with the repo and its dependencies. +这会启动一个 Docker 容器，并将你的本地仓库挂载到 `/mnt/local_unstructured`。该 Docker 镜像可让你无需担心操作系统与仓库及其依赖的兼容性问题即可进行开发。 ## :clap: Quick Tour -### Documentation -For more comprehensive documentation, visit https://docs.unstructured.io . You can also learn -more about our other products on the documentation page, including our SaaS API. +### 文档 -Here are a few pages from the [Open Source documentation page](https://docs.unstructured.io/open-source/introduction/overview) -that are helpful for new users to review: +更完整的文档请访问 https://docs.unstructured.io。你也可以在文档页面了解我们的其他产品，包括我们的 SaaS API。 + +以下是[开源文档页面](https://docs.unstructured.io/open-source/introduction/overview) 中对新用户有帮助的几页： - [Quick Start](https://docs.unstructured.io/open-source/introduction/quick-start) - [Using the `unstructured` open source package](https://docs.unstructured.io/open-source/core-functionality/overview) @@ -223,10 +217,11 @@ that are helpful for new users to review: - [Integrations](https://docs.unstructured.io/open-source/integrations) -### PDF Document Parsing Example -The following examples show how to get started with the `unstructured` library. The easiest way to parse a document in unstructured is to use the `partition` function. If you use `partition` function, `unstructured` will detect the file type and route it to the appropriate file-specific partitioning function. If you are using the `partition` function, you may need to install additional dependencies per doc type. -For example, to install docx dependencies you need to run `pip install "unstructured[docx]"`. -See our [installation guide](https://docs.unstructured.io/open-source/installation/full-installation) for more details. +### PDF 文档解析示例 + +以下示例展示如何开始使用 `unstructured` 库。在 unstructured 中解析文档的最简单方式是使用 `partition` 函数。若使用 `partition` 函数，`unstructured` 会检测文件类型并将其路由到相应的文件专用分区（partitioning）函数。若使用 `partition` 函数，你可能需要按文档类型安装额外依赖。 +例如，要安装 docx 依赖，你需要运行 `pip install "unstructured[docx]"`。 +更多细节请参阅我们的[安装指南](https://docs.unstructured.io/open-source/installation/full-installation)。 ```python from unstructured.partition.auto import partition @@ -234,8 +229,7 @@ from unstructured.partition.auto import partition elements = partition("example-docs/layout-parser-paper.pdf") ``` -Run `print("\n\n".join([str(el) for el in elements]))` to get a string representation of the -output, which looks like: +运行 `print("\n\n".join([str(el) for el in elements]))` 可得到输出的字符串表示，形式如下： ``` @@ -267,27 +261,26 @@ Deep Learning(DL)-based approaches are the state-of-the-art for a wide range of including document image classification [11, ``` -See the [partitioning](https://docs.unstructured.io/open-source/core-functionality/partitioning) -section in our documentation for a full list of options and instructions on how to use -file-specific partitioning functions. +请参阅我们文档中的 [partitioning](https://docs.unstructured.io/open-source/core-functionality/partitioning) +章节，了解完整选项列表以及如何使用特定于文件的分区（partitioning）函数的说明。 -## :guardsman: Security Policy +## :guardsman: 安全策略 -See our [security policy](https://github.com/Unstructured-IO/unstructured/security/policy) for -information on how to report security vulnerabilities. +请参阅我们的 [安全策略](https://github.com/Unstructured-IO/unstructured/security/policy) for +了解如何报告安全漏洞的相关信息。 -## :bug: Reporting Bugs +## :bug: 报告 Bug -Encountered a bug? Please create a new [GitHub issue](https://github.com/Unstructured-IO/unstructured/issues/new/choose) and use our bug report template to describe the problem. To help us diagnose the issue, use the `python scripts/collect_env.py` command to gather your system's environment information and include it in your report. Your assistance helps us continuously improve our software - thank you! +遇到了 bug？请创建新的 [GitHub issue](https://github.com/Unstructured-IO/unstructured/issues/new/choose) and use our bug 报告模板描述问题。为帮助我们诊断问题，请使用 `python scripts/collect_env.py` 命令收集您系统的环境信息并附在报告中。您的帮助有助于我们持续改进软件——感谢！ -## :books: Learn more +## :books: 了解更多 -| Section | Description | +| 章节 | 说明 | |-|-| -| [Company Website](https://unstructured.io) | Unstructured.io product and company info | -| [Documentation](https://docs.unstructured.io/) | Full API documentation | -| [Batch Processing](https://github.com/Unstructured-IO/unstructured-ingest) | Ingesting batches of documents through Unstructured | +| [Company Website](https://unstructured.io) | Unstructured.io 产品与公司信息 | +| [Documentation](https://docs.unstructured.io/) | 完整 API 文档 | +| [Batch Processing](https://github.com/Unstructured-IO/unstructured-ingest) | 通过 Unstructured 批量摄入文档 | -## :chart_with_upwards_trend: Analytics +## :chart_with_upwards_trend: 分析 -Telemetry is **off by default**. To opt in, set `UNSTRUCTURED_TELEMETRY_ENABLED=true` (or `=1`) before importing `unstructured`. To opt out, set `DO_NOT_TRACK` or `SCARF_NO_ANALYTICS` to any non-empty value (e.g. `true`, `1`, `yes`, `false`, `0`—any non-empty string opts out); opt-out takes precedence. Unset the variable or leave it empty if you do not want to opt out. See our [Privacy Policy](https://unstructured.io/privacy-policy). +遥测（Telemetry）默认**关闭**。要选择加入（opt in），请在导入 `unstructured` 之前设置 `UNSTRUCTURED_TELEMETRY_ENABLED=true`（或 `=1`）。要选择退出（opt out），请将 `DO_NOT_TRACK` 或 `SCARF_NO_ANALYTICS` 设置为任意非空值（例如 `true`、`1`、`yes`、`false`、`0`——任意非空字符串均表示退出）；退出设置优先。若不想退出，请取消设置该变量或将其留空。请参阅我们的 [隐私政策](https://unstructured.io/privacy-policy).