diff --git a/README.md b/README.md index 838c930..68f42ea 100644 --- a/README.md +++ b/README.md @@ -1,122 +1,127 @@ + +> [!NOTE] +> 本文档由 WeHub 基于上游 README 翻译整理，属于社区翻译，非官方中文文档。 +> [English](./README.en.md) · [原始项目](https://github.com/IDEA-Research/Grounded-Segment-Anything) · [上游 README](https://github.com/IDEA-Research/Grounded-Segment-Anything/blob/HEAD/README.md) +> 原作者、版权与许可证归属以原始项目及本仓库 LICENSE 文件为准。 + ![](./assets/Grounded-SAM_logo.png) # Grounded-Segment-Anything [![YouTube](https://badges.aleen42.com/src/youtube.svg)](https://youtu.be/oEQYStnF2l8) [![Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/roboflow-ai/notebooks/blob/main/notebooks/automated-dataset-annotation-and-evaluation-with-grounding-dino-and-sam.ipynb) [![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://github.com/camenduru/grounded-segment-anything-colab) [![HuggingFace Space](https://img.shields.io/badge/🤗-HuggingFace%20Space-cyan.svg)](https://huggingface.co/spaces/IDEA-Research/Grounded-SAM) [![Replicate](https://replicate.com/cjwbw/grounded-recognize-anything/badge)](https://replicate.com/cjwbw/grounded-recognize-anything) [![ModelScope Official Demo](https://img.shields.io/badge/ModelScope-Official%20Demo-important)](https://modelscope.cn/studios/tuofeilunhifi/Grounded-Segment-Anything/summary) [![Huggingface Demo by Community](https://img.shields.io/badge/Huggingface-Demo%20by%20Community-red)](https://huggingface.co/spaces/yizhangliu/Grounded-Segment-Anything) [![Stable-Diffusion WebUI](https://img.shields.io/badge/Stable--Diffusion-WebUI%20by%20Community-critical)](https://github.com/continue-revolution/sd-webui-segment-anything) [![Jupyter Notebook Demo](https://img.shields.io/badge/Demo-Jupyter%20Notebook-informational)](./grounded_sam.ipynb) [![Static Badge](https://img.shields.io/badge/GroundingDINO-arXiv-blue)](https://arxiv.org/abs/2303.05499) [![Static Badge](https://img.shields.io/badge/Segment_Anything-arXiv-blue)](https://arxiv.org/abs/2304.02643) [![Static Badge](https://img.shields.io/badge/Grounded_SAM-arXiv-blue)](https://arxiv.org/abs/2401.14159) -We plan to create a very interesting demo by combining [Grounding DINO](https://github.com/IDEA-Research/GroundingDINO) and [Segment Anything](https://github.com/facebookresearch/segment-anything) which aims to detect and segment anything with text inputs! And we will continue to improve it and create more interesting demos based on this foundation. And we have already released an overall technical report about our project on arXiv, please check [Grounded SAM: Assembling Open-World Models for Diverse Visual Tasks](https://arxiv.org/abs/2401.14159) for more details. +我们计划通过结合 [Grounding DINO](https://github.com/IDEA-Research/GroundingDINO) 与 [Segment Anything](https://github.com/facebookresearch/segment-anything)，打造一个非常有趣的演示：用文本输入即可检测并分割任意目标！我们将在此基础上持续改进，并打造更多有趣的演示。我们已在 arXiv 发布了关于本项目的整体技术报告，详情请参阅 [Grounded SAM: Assembling Open-World Models for Diverse Visual Tasks](https://arxiv.org/abs/2401.14159)。 -- 🔥 **[Grounded SAM 2](https://github.com/IDEA-Research/Grounded-SAM-2)** is released now, which combines Grounding DINO with [SAM 2](https://github.com/facebookresearch/segment-anything-2) for any object tracking in open-world scenarios. -- 🔥 **[Grounding DINO 1.5](https://github.com/IDEA-Research/Grounding-DINO-1.5-API)** is released now, which is IDEA Research's **Most Capable** Open-World Object Detection Model! -- 🔥 **[Grounding DINO](https://arxiv.org/abs/2303.05499)** and **[Grounded SAM](https://arxiv.org/abs/2401.14159)** are now supported in Huggingface. For more convenient use, you can refer to [this documentation](https://huggingface.co/docs/transformers/model_doc/grounding-dino) +- 🔥 **[Grounded SAM 2](https://github.com/IDEA-Research/Grounded-SAM-2)** 现已发布，它将 Grounding DINO 与 [SAM 2](https://github.com/facebookresearch/segment-anything-2) 结合，可在开放世界场景中跟踪任意目标。 +- 🔥 **[Grounding DINO 1.5](https://github.com/IDEA-Research/Grounding-DINO-1.5-API)** 现已发布，这是 IDEA Research 能力最强的开放世界目标检测模型！ +- 🔥 **[Grounding DINO](https://arxiv.org/abs/2303.05499)** 与 **[Grounded SAM](https://arxiv.org/abs/2401.14159)** 现已支持 Huggingface。为便于使用，可参考 [此文档](https://huggingface.co/docs/transformers/model_doc/grounding-dino) -We are very willing to **help everyone share and promote new projects** based on Segment-Anything, Please check out here for more amazing demos and works in the community: [Highlight Extension Projects](#highlighted-projects). You can submit a new issue (with `project` tag) or a new pull request to add new project's links. +我们非常愿意**帮助每个人分享并推广**基于 Segment-Anything 的新项目。更多社区中的精彩演示与作品请见：[Highlight Extension Projects](#highlighted-projects)。你可以提交新 issue（带 `project` 标签）或新 pull request 来添加新项目链接。 ![](./assets/grounded_sam_new_demo_image.png) ![](./assets/ram_grounded_sam_new.png) -**🍄 Why Building this Project?** +**🍄 为什么要做这个项目？** -The **core idea** behind this project is to **combine the strengths of different models in order to build a very powerful pipeline for solving complex problems**. And it's worth mentioning that this is a workflow for combining strong expert models, where **all parts can be used separately or in combination, and can be replaced with any similar but different models (like replacing Grounding DINO with GLIP or other detectors / replacing Stable-Diffusion with ControlNet or GLIGEN/ Combining with ChatGPT)**. +本项目的**核心思路**是**组合不同模型的优势，构建一条非常强大的流水线以解决复杂问题**。值得一提的是，这是一套组合强专家模型的工作流，其中**各部分既可单独使用，也可组合使用，并可替换为任何类似但不同的模型**（例如用 GLIP 或其他检测器替换 Grounding DINO / 用 ControlNet 或 GLIGEN 替换 Stable-Diffusion / 与 ChatGPT 结合）。 -**🍇 Updates** -- **`2024/01/26`** We have released a comprehensive technical report about our project on arXiv, please check [Grounded SAM: Assembling Open-World Models for Diverse Visual Tasks](https://arxiv.org/abs/2401.14159) for more details. And we are profoundly grateful for the contributions of all the contributors in this project. -- **`2023/12/17`** Support [Grounded-RepViT-SAM](https://github.com/IDEA-Research/Grounded-Segment-Anything/tree/main/EfficientSAM#run-grounded-repvit-sam-demo) demo, thanks a lot for their great work! -- **`2023/12/16`** Support [Grounded-Edge-SAM](https://github.com/IDEA-Research/Grounded-Segment-Anything/tree/main/EfficientSAM#run-grounded-edge-sam-demo) demo, thanks a lot for their great work! -- **`2023/12/10`** Support [Grounded-Efficient-SAM](https://github.com/IDEA-Research/Grounded-Segment-Anything/tree/main/EfficientSAM#run-grounded-efficient-sam-demo) demo, thanks a lot for their great work! -- **`2023/11/24`** Release [RAM++](https://arxiv.org/abs/2310.15200), which is the next generation of RAM. RAM++ can recognize any category with high accuracy, including both predefined common categories and diverse open-set categories. -- **`2023/11/23`** Release our newly proposed visual prompt counting model [T-Rex](https://github.com/IDEA-Research/T-Rex). The introduction [Video](https://www.youtube.com/watch?v=engIEhZogAQ) and [Demo](https://deepdataspace.com/playground/ivp) is available in [DDS](https://github.com/IDEA-Research/deepdataspace) now. -- **`2023/07/25`** Support [Light-HQ-SAM](https://github.com/SysCV/sam-hq) in [EfficientSAM](./EfficientSAM/), credits to [Mingqiao Ye](https://github.com/ymq2017) and [Lei Ke](https://github.com/lkeab), thanks a lot for their great work! -- **`2023/07/14`** Combining **Grounding-DINO-B** with [SAM-HQ](https://github.com/SysCV/sam-hq) achieves **49.6 mean AP** in [Segmentation in the Wild](https://eval.ai/web/challenges/challenge-page/1931/overview) competition zero-shot track, surpassing Grounded-SAM by **3.6 mean AP**, thanks for their great work! -- **`2023/06/28`** Combining Grounding-DINO with Efficient SAM variants including [FastSAM](https://github.com/CASIA-IVA-Lab/FastSAM) and [MobileSAM](https://github.com/ChaoningZhang/MobileSAM) in [EfficientSAM](./EfficientSAM/) for faster annotating, thanks a lot for their great work! -- **`2023/06/20`** By combining **Grounding-DINO-L** with **SAM-ViT-H**, Grounded-SAM achieves 46.0 mean AP in [Segmentation in the Wild](https://eval.ai/web/challenges/challenge-page/1931/overview) competition zero-shot track on [CVPR 2023 workshop](https://computer-vision-in-the-wild.github.io/cvpr-2023/), surpassing [UNINEXT (CVPR 2023)](https://github.com/MasterBin-IIAU/UNINEXT) by about **4 mean AP**. -- **`2023/06/16`** Release [RAM-Grounded-SAM Replicate Online Demo](https://replicate.com/cjwbw/ram-grounded-sam). Thanks a lot to [Chenxi](https://chenxwh.github.io/) for providing this nice demo 🌹. -- **`2023/06/14`** Support [RAM-Grounded-SAM & SAM-HQ](./automatic_label_ram_demo.py) and update [Simple Automatic Label Demo](./automatic_label_ram_demo.py) to support [RAM](https://github.com/OPPOMKLab/recognize-anything), setting up a strong automatic annotation pipeline. -- **`2023/06/13`** Checkout the [Autodistill: Train YOLOv8 with ZERO Annotations](https://youtu.be/gKTYMfwPo4M) tutorial to learn how to use Grounded-SAM + [Autodistill](https://github.com/autodistill/autodistill) for automated data labeling and real-time model training. -- **`2023/06/13`** Support [SAM-HQ](https://github.com/SysCV/sam-hq) in [Grounded-SAM Demo](#running_man-grounded-sam-detect-and-segment-everything-with-text-prompt) for higher quality prediction. -- **`2023/06/12`** Support [RAM-Grounded-SAM](#label-grounded-sam-with-ram-or-tag2text-for-automatic-labeling) for strong automatic labeling pipeline! Thanks for [Recognize-Anything](https://github.com/OPPOMKLab/recognize-anything). -- **`2023/06/01`** Our Grounded-SAM has been accepted to present a **demo** at [ICCV 2023](https://iccv2023.thecvf.com/)! See you in Paris! -- **`2023/05/23`**: Support `Image-Referring-Segment`, `Audio-Referring-Segment` and `Text-Referring-Segment` in [ImageBind-SAM](./playground/ImageBind_SAM/). -- **`2023/05/03`**: Checkout the [Automated Dataset Annotation and Evaluation with GroundingDINO and SAM](https://colab.research.google.com/github/roboflow-ai/notebooks/blob/main/notebooks/automated-dataset-annotation-and-evaluation-with-grounding-dino-and-sam.ipynb) which is an amazing tutorial on automatic labeling! Thanks a lot for [Piotr Skalski](https://github.com/SkalskiP) and [Roboflow](https://github.com/roboflow/notebooks)! +**🍇 更新** +- **`2024/01/26`** 我们已在 arXiv 发布关于本项目的综合技术报告，详情请参阅 [Grounded SAM: Assembling Open-World Models for Diverse Visual Tasks](https://arxiv.org/abs/2401.14159)。衷心感谢本项目所有贡献者的付出。 +- **`2023/12/17`** 支持 [Grounded-RepViT-SAM](https://github.com/IDEA-Research/Grounded-Segment-Anything/tree/main/EfficientSAM#run-grounded-repvit-sam-demo) 演示，非常感谢他们的出色工作！ +- **`2023/12/16`** 支持 [Grounded-Edge-SAM](https://github.com/IDEA-Research/Grounded-Segment-Anything/tree/main/EfficientSAM#run-grounded-edge-sam-demo) 演示，非常感谢他们的出色工作！ +- **`2023/12/10`** 支持 [Grounded-Efficient-SAM](https://github.com/IDEA-Research/Grounded-Segment-Anything/tree/main/EfficientSAM#run-grounded-efficient-sam-demo) 演示，非常感谢他们的出色工作！ +- **`2023/11/24`** 发布 [RAM++](https://arxiv.org/abs/2310.15200),，这是 RAM 的下一代版本。RAM++ 能以高精度识别任意类别，包括预定义的常见类别和多样化的开放集类别。 +- **`2023/11/23`** 发布我们新提出的视觉提示计数模型 [T-Rex](https://github.com/IDEA-Research/T-Rex).。[介绍视频](https://www.youtube.com/watch?v=engIEhZogAQ) 与 [演示](https://deepdataspace.com/playground/ivp) 现已在 [DDS](https://github.com/IDEA-Research/deepdataspace) 提供。 +- **`2023/07/25`** 在 [EfficientSAM](./EfficientSAM/) 中支持 [Light-HQ-SAM](https://github.com/SysCV/sam-hq)，致谢 [Mingqiao Ye](https://github.com/ymq2017) 与 [Lei Ke](https://github.com/lkeab),，非常感谢他们的出色工作！ +- **`2023/07/14`** 将 **Grounding-DINO-B** 与 [SAM-HQ](https://github.com/SysCV/sam-hq) 结合，在 [Segmentation in the Wild](https://eval.ai/web/challenges/challenge-page/1931/overview) 竞赛零样本赛道取得 **49.6 mean AP**，较 Grounded-SAM 提升 **3.6 mean AP**，感谢他们的出色工作！ +- **`2023/06/28`** 在 [EfficientSAM](./EfficientSAM/) 中将 Grounding-DINO 与包括 [FastSAM](https://github.com/CASIA-IVA-Lab/FastSAM) 与 [MobileSAM](https://github.com/ChaoningZhang/MobileSAM) 在内的 Efficient SAM 变体结合，以实现更快标注，非常感谢他们的出色工作！ +- **`2023/06/20`** 将 **Grounding-DINO-L** 与 **SAM-ViT-H** 结合，Grounded-SAM 在 [Segmentation in the Wild](https://eval.ai/web/challenges/challenge-page/1931/overview) 竞赛零样本赛道、[CVPR 2023 workshop](https://computer-vision-in-the-wild.github.io/cvpr-2023/), 上取得 46.0 mean AP，较 [UNINEXT (CVPR 2023)](https://github.com/MasterBin-IIAU/UNINEXT) 提升约 **4 mean AP**。 +- **`2023/06/16`** 发布 [RAM-Grounded-SAM Replicate Online Demo](https://replicate.com/cjwbw/ram-grounded-sam).，非常感谢 [Chenxi](https://chenxwh.github.io/) 提供这一精彩演示 🌹。 +- **`2023/06/14`** 支持 [RAM-Grounded-SAM & SAM-HQ](./automatic_label_ram_demo.py)，并更新 [Simple Automatic Label Demo](./automatic_label_ram_demo.py) 以支持 [RAM](https://github.com/OPPOMKLab/recognize-anything),，搭建强大的自动标注流水线。 +- **`2023/06/13`** 查看 [Autodistill: Train YOLOv8 with ZERO Annotations](https://youtu.be/gKTYMfwPo4M) 教程，了解如何使用 Grounded-SAM + [Autodistill](https://github.com/autodistill/autodistill) 进行自动数据标注与实时模型训练。 +- **`2023/06/13`** 在 [Grounded-SAM Demo](#running_man-grounded-sam-detect-and-segment-everything-with-text-prompt) 中支持 [SAM-HQ](https://github.com/SysCV/sam-hq)，以获得更高质量的预测。 +- **`2023/06/12`** 支持 [RAM-Grounded-SAM](#label-grounded-sam-with-ram-or-tag2text-for-automatic-labeling)，打造强大的自动标注流水线！感谢 [Recognize-Anything](https://github.com/OPPOMKLab/recognize-anything). +- **`2023/06/01`** 我们的 Grounded-SAM 已获录用，将在 [ICCV 2023](https://iccv2023.thecvf.com/)! 以**演示**形式展示。巴黎见！ +- **`2023/05/23`**：在 [ImageBind-SAM](./playground/ImageBind_SAM/) 中支持 `Image-Referring-Segment`、`Audio-Referring-Segment` 与 `Text-Referring-Segment`。 +- **`2023/05/03`**：查看 [Automated Dataset Annotation and Evaluation with GroundingDINO and SAM](https://colab.research.google.com/github/roboflow-ai/notebooks/blob/main/notebooks/automated-dataset-annotation-and-evaluation-with-grounding-dino-and-sam.ipynb)，这是一篇关于自动标注的精彩教程！非常感谢 [Piotr Skalski](https://github.com/SkalskiP) 与 [Roboflow](https://github.com/roboflow/notebooks)! - -## Table of Contents +## 目录 - [Grounded-Segment-Anything](#grounded-segment-anything) - - [Preliminary Works](#preliminary-works) - - [Highlighted Projects](#highlighted-projects) -- [Installation](#installation) - - [Install with Docker](#install-with-docker) - - [Install locally](#install-without-docker) -- [Grounded-SAM Playground](#grounded-sam-playground) - - [Step-by-Step Notebook Demo](#open_book-step-by-step-notebook-demo) - - [GroundingDINO: Detect Everything with Text Prompt](#running_man-groundingdino-detect-everything-with-text-prompt) - - [Grounded-SAM: Detect and Segment Everything with Text Prompt](#running_man-grounded-sam-detect-and-segment-everything-with-text-prompt) - - [Grounded-SAM with Inpainting: Detect, Segment and Generate Everything with Text Prompt](#skier-grounded-sam-with-inpainting-detect-segment-and-generate-everything-with-text-prompt) - - [Grounded-SAM and Inpaint Gradio APP](#golfing-grounded-sam-and-inpaint-gradio-app) - - [Grounded-SAM with RAM or Tag2Text for Automatic Labeling](#label-grounded-sam-with-ram-or-tag2text-for-automatic-labeling) - - [Grounded-SAM with BLIP & ChatGPT for Automatic Labeling](#robot-grounded-sam-with-blip-for-automatic-labeling) - - [Grounded-SAM with Whisper: Detect and Segment Anything with Audio](#open_mouth-grounded-sam-with-whisper-detect-and-segment-anything-with-audio) - - [Grounded-SAM ChatBot with Visual ChatGPT](#speech_balloon-grounded-sam-chatbot-demo) - - [Grounded-SAM with OSX for 3D Whole-Body Mesh Recovery](#man_dancing-run-grounded-segment-anything--osx-demo) - - [Grounded-SAM with VISAM for Tracking and Segment Anything](#man_dancing-run-grounded-segment-anything--visam-demo) - - [Interactive Fashion-Edit Playground: Click for Segmentation And Editing](#dancers-interactive-editing) - - [Interactive Human-face Editing Playground: Click And Editing Human Face](#dancers-interactive-editing) - - [3D Box Via Segment Anything](#camera-3d-box-via-segment-anything) - - [Playground: More Interesting and Imaginative Demos with Grounded-SAM](./playground/) - - [DeepFloyd: Image Generation with Text Prompt](./playground/DeepFloyd/) - - [PaintByExample: Exemplar-based Image Editing with Diffusion Models](./playground/PaintByExample/) - - [LaMa: Resolution-robust Large Mask Inpainting with Fourier Convolutions](./playground/LaMa/) - - [RePaint: Inpainting using Denoising Diffusion Probabilistic Models](./playground/RePaint/) - - [ImageBind with SAM: Segment with Different Modalities](./playground/ImageBind_SAM/) - - [Efficient SAM Series for Faster Annotation](./EfficientSAM/) - - [Grounded-FastSAM Demo](https://github.com/IDEA-Research/Grounded-Segment-Anything/tree/main/EfficientSAM#run-grounded-fastsam-demo) - - [Grounded-MobileSAM Demo](https://github.com/IDEA-Research/Grounded-Segment-Anything/tree/main/EfficientSAM#run-grounded-mobilesam-demo) - - [Grounded-Light-HQSAM Demo](https://github.com/IDEA-Research/Grounded-Segment-Anything/tree/main/EfficientSAM#run-grounded-light-hqsam-demo) - - [Grounded-Efficient-SAM Demo](https://github.com/IDEA-Research/Grounded-Segment-Anything/tree/main/EfficientSAM#run-grounded-efficient-sam-demo) - - [Grounded-Edge-SAM Demo](https://github.com/IDEA-Research/Grounded-Segment-Anything/tree/main/EfficientSAM#run-grounded-edge-sam-demo) - - [Grounded-RepViT-SAM Demo](https://github.com/IDEA-Research/Grounded-Segment-Anything/tree/main/EfficientSAM#run-grounded-repvit-sam-demo) -- [Citation](#citation) + - [前期工作](#preliminary-works) + - [精选项目](#highlighted-projects) +- [安装](#installation) + - [使用 Docker 安装](#install-with-docker) + - [本地安装](#install-without-docker) +- [Grounded-SAM 演示区](#grounded-sam-playground) + - [分步 Notebook 演示](#open_book-step-by-step-notebook-demo) + - [GroundingDINO：使用文本提示检测一切](#running_man-groundingdino-detect-everything-with-text-prompt) + - [Grounded-SAM：使用文本提示检测并分割一切](#running_man-grounded-sam-detect-and-segment-everything-with-text-prompt) + - [带修复（Inpainting）的 Grounded-SAM：使用文本提示检测、分割并生成一切](#skier-grounded-sam-with-inpainting-detect-segment-and-generate-everything-with-text-prompt) + - [Grounded-SAM 与 Inpaint Gradio 应用](#golfing-grounded-sam-and-inpaint-gradio-app) + - [Grounded-SAM 配合 RAM 或 Tag2Text 实现自动标注](#label-grounded-sam-with-ram-or-tag2text-for-automatic-labeling) + - [Grounded-SAM 配合 BLIP 与 ChatGPT 实现自动标注](#robot-grounded-sam-with-blip-for-automatic-labeling) + - [Grounded-SAM 配合 Whisper：使用音频检测并分割任意对象](#open_mouth-grounded-sam-with-whisper-detect-and-segment-anything-with-audio) + - [配合 Visual ChatGPT 的 Grounded-SAM 聊天机器人](#speech_balloon-grounded-sam-chatbot-demo) + - [Grounded-SAM 配合 OSX 实现 3D 全身网格恢复](#man_dancing-run-grounded-segment-anything--osx-demo) + - [Grounded-SAM 配合 VISAM 实现跟踪与分割任意对象](#man_dancing-run-grounded-segment-anything--visam-demo) + - [交互式时尚编辑演示区：点击进行分割与编辑](#dancers-interactive-editing) + - [交互式人脸编辑演示区：点击并编辑人脸](#dancers-interactive-editing) + - [通过 Segment Anything 生成 3D 框](#camera-3d-box-via-segment-anything) + - [演示区：更多有趣且富有想象力的 Grounded-SAM 演示](./playground/) + - [DeepFloyd：使用文本提示生成图像](./playground/DeepFloyd/) + - [PaintByExample：基于扩散模型的示例驱动图像编辑](./playground/PaintByExample/) + - [LaMa：基于傅里叶卷积的分辨率鲁棒大掩码修复](./playground/LaMa/) + - [RePaint：使用去噪扩散概率模型进行修复](./playground/RePaint/) + - [ImageBind 配合 SAM：跨模态分割](./playground/ImageBind_SAM/) + - [高效 SAM 系列，加速标注](./EfficientSAM/) + - [Grounded-FastSAM 演示](https://github.com/IDEA-Research/Grounded-Segment-Anything/tree/main/EfficientSAM#run-grounded-fastsam-demo) + - [Grounded-MobileSAM 演示](https://github.com/IDEA-Research/Grounded-Segment-Anything/tree/main/EfficientSAM#run-grounded-mobilesam-demo) + - [Grounded-Light-HQSAM 演示](https://github.com/IDEA-Research/Grounded-Segment-Anything/tree/main/EfficientSAM#run-grounded-light-hqsam-demo) + - [Grounded-Efficient-SAM 演示](https://github.com/IDEA-Research/Grounded-Segment-Anything/tree/main/EfficientSAM#run-grounded-efficient-sam-demo) + - [Grounded-Edge-SAM 演示](https://github.com/IDEA-Research/Grounded-Segment-Anything/tree/main/EfficientSAM#run-grounded-edge-sam-demo) + - [Grounded-RepViT-SAM 演示](https://github.com/IDEA-Research/Grounded-Segment-Anything/tree/main/EfficientSAM#run-grounded-repvit-sam-demo) +- [引用](#citation) -## Preliminary Works +## 前期工作 -Here we provide some background knowledge that you may need to know before trying the demos. +在尝试演示之前，这里提供一些你可能需要了解的背景知识。
-| Title | Intro | Description | Links | +| 标题 | 简介 | 描述 | 链接 | |:----:|:----:|:----:|:----:| -| [Segment-Anything](https://arxiv.org/abs/2304.02643) | ![](https://github.com/facebookresearch/segment-anything/blob/main/assets/model_diagram.png?raw=true) | A strong foundation model aims to segment everything in an image, which needs prompts (as boxes/points/text) to generate masks | [[Github](https://github.com/facebookresearch/segment-anything)]
[[Page](https://segment-anything.com/)]
[[Demo](https://segment-anything.com/demo)] | -| [Grounding DINO](https://arxiv.org/abs/2303.05499) | ![](https://github.com/IDEA-Research/GroundingDINO/blob/main/.asset/hero_figure.png?raw=True) | A strong zero-shot detector which is capable of to generate high quality boxes and labels with free-form text. | [[Github](https://github.com/IDEA-Research/GroundingDINO)]
[[Demo](https://huggingface.co/spaces/ShilongLiu/Grounding_DINO_demo)] | -| [OSX](http://arxiv.org/abs/2303.16160) | ![](https://github.com/IDEA-Research/OSX/blob/main/assets/demo_video.gif?raw=True) | A strong and efficient one-stage motion capture method to generate high quality 3D human mesh from monucular image. OSX also releases a large-scale upper-body dataset UBody for a more accurate reconstrution in the upper-body scene. | [[Github](https://github.com/IDEA-Research/OSX)]
[[Page](https://osx-ubody.github.io/)]
[[Video](https://osx-ubody.github.io/)]
[[Data](https://docs.google.com/forms/d/e/1FAIpQLSehgBP7wdn_XznGAM2AiJPiPLTqXXHw5uX9l7qeQ1Dh9HoO_A/viewform)] | -| [Stable-Diffusion](https://arxiv.org/abs/2112.10752) | ![](https://github.com/CompVis/stable-diffusion/blob/main/assets/stable-samples/txt2img/merged-0006.png?raw=True) | A super powerful open-source latent text-to-image diffusion model | [[Github](https://github.com/CompVis/stable-diffusion)]
[[Page](https://ommer-lab.com/research/latent-diffusion-models/)] | -| [RAM++](https://arxiv.org/abs/2310.15200) | ![](https://github.com/xinyu1205/recognize-anything/blob/main/images/ram_plus_compare.jpg) | RAM++ is the next generation of RAM, which can recognize any category with high accuracy. | [[Github](https://github.com/OPPOMKLab/recognize-anything)] | -| [RAM](https://recognize-anything.github.io/) | ![](https://github.com/xinyu1205/Tag2Text/raw/main/images/localization_and_recognition.jpg) | RAM is an image tagging model, which can recognize any common category with high accuracy. | [[Github](https://github.com/OPPOMKLab/recognize-anything)]
[[Demo](https://huggingface.co/spaces/xinyu1205/Recognize_Anything-Tag2Text)] | -| [BLIP](https://arxiv.org/abs/2201.12086) | ![](https://github.com/salesforce/LAVIS/raw/main/docs/_static/logo_final.png) | A wonderful language-vision model for image understanding. | [[GitHub](https://github.com/salesforce/LAVIS)] | -| [Visual ChatGPT](https://arxiv.org/abs/2303.04671) | ![](https://github.com/microsoft/TaskMatrix/raw/main/assets/figure.jpg) | A wonderful tool that connects ChatGPT and a series of Visual Foundation Models to enable sending and receiving images during chatting. | [[Github](https://github.com/microsoft/TaskMatrix)]
[[Demo](https://huggingface.co/spaces/microsoft/visual_chatgpt)] | -| [Tag2Text](https://tag2text.github.io/) | ![](https://github.com/xinyu1205/Tag2Text/raw/main/images/tag2text_framework.png) | An efficient and controllable vision-language model which can simultaneously output superior image captioning and image tagging. | [[Github](https://github.com/OPPOMKLab/recognize-anything)]
[[Demo](https://huggingface.co/spaces/xinyu1205/Tag2Text)] | -| [VoxelNeXt](https://arxiv.org/abs/2303.11301) | ![](https://github.com/dvlab-research/VoxelNeXt/raw/master/docs/sequence-v2.gif) | A clean, simple, and fully-sparse 3D object detector, which predicts objects directly upon sparse voxel features. | [[Github](https://github.com/dvlab-research/VoxelNeXt)] +| [Segment-Anything](https://arxiv.org/abs/2304.02643) | ![](https://github.com/facebookresearch/segment-anything/blob/main/assets/model_diagram.png?raw=true) | 一个强大的基础模型，旨在分割图像中的任意对象，需要提示（如框/点/文本）来生成掩码 | [[Github](https://github.com/facebookresearch/segment-anything)]
[[Page](https://segment-anything.com/)]
[[Demo](https://segment-anything.com/demo)] | +| [Grounding DINO](https://arxiv.org/abs/2303.05499) | ![](https://github.com/IDEA-Research/GroundingDINO/blob/main/.asset/hero_figure.png?raw=True) | 一个强大的零样本检测器，能够使用自由形式文本生成高质量的框和标签。 | [[Github](https://github.com/IDEA-Research/GroundingDINO)]
[[Demo](https://huggingface.co/spaces/ShilongLiu/Grounding_DINO_demo)] | +| [OSX](http://arxiv.org/abs/2303.16160) | ![](https://github.com/IDEA-Research/OSX/blob/main/assets/demo_video.gif?raw=True) | 一种强大且高效的单阶段动作捕捉方法，可从单目图像生成高质量的 3D 人体网格。OSX 还发布了大规模上半身数据集 UBody，以在上半身场景中实现更准确的重建。 | [[Github](https://github.com/IDEA-Research/OSX)]
[[Page](https://osx-ubody.github.io/)]
[[Video](https://osx-ubody.github.io/)]
[[Data](https://docs.google.com/forms/d/e/1FAIpQLSehgBP7wdn_XznGAM2AiJPiPLTqXXHw5uX9l7qeQ1Dh9HoO_A/viewform)] | +| [Stable-Diffusion](https://arxiv.org/abs/2112.10752) | ![](https://github.com/CompVis/stable-diffusion/blob/main/assets/stable-samples/txt2img/merged-0006.png?raw=True) | 一个超级强大的开源潜空间文本到图像扩散模型 | [[Github](https://github.com/CompVis/stable-diffusion)]
[[Page](https://ommer-lab.com/research/latent-diffusion-models/)] | +| [RAM++](https://arxiv.org/abs/2310.15200) | ![](https://github.com/xinyu1205/recognize-anything/blob/main/images/ram_plus_compare.jpg) | RAM++ 是 RAM 的下一代版本，能够以高精度识别任意类别。 | [[Github](https://github.com/OPPOMKLab/recognize-anything)] | +| [RAM](https://recognize-anything.github.io/) | ![](https://github.com/xinyu1205/Tag2Text/raw/main/images/localization_and_recognition.jpg) | RAM 是一个图像标注模型，能够以高精度识别任意常见类别。 | [[Github](https://github.com/OPPOMKLab/recognize-anything)]
[[Demo](https://huggingface.co/spaces/xinyu1205/Recognize_Anything-Tag2Text)] | +| [BLIP](https://arxiv.org/abs/2201.12086) | ![](https://github.com/salesforce/LAVIS/raw/main/docs/_static/logo_final.png) | 一个出色的语言-视觉模型，用于图像理解。 | [[GitHub](https://github.com/salesforce/LAVIS)] | +| [Visual ChatGPT](https://arxiv.org/abs/2303.04671) | ![](https://github.com/microsoft/TaskMatrix/raw/main/assets/figure.jpg) | 一个出色的工具，将 ChatGPT 与一系列视觉基础模型（Visual Foundation Models）连接起来，使聊天过程中能够发送和接收图像。 | [[Github](https://github.com/microsoft/TaskMatrix)]
[[Demo](https://huggingface.co/spaces/microsoft/visual_chatgpt)] | +| [Tag2Text](https://tag2text.github.io/) | ![](https://github.com/xinyu1205/Tag2Text/raw/main/images/tag2text_framework.png) | 一个高效且可控的视觉-语言模型，能够同时输出优质的图像描述和图像标注。 | [[Github](https://github.com/OPPOMKLab/recognize-anything)]
[[Demo](https://huggingface.co/spaces/xinyu1205/Tag2Text)] | +| [VoxelNeXt](https://arxiv.org/abs/2303.11301) | ![](https://github.com/dvlab-research/VoxelNeXt/raw/master/docs/sequence-v2.gif) | 一个简洁、简单且完全稀疏的 3D 目标检测器，直接在稀疏体素特征上预测目标。 | [[Github](https://github.com/dvlab-research/VoxelNeXt)]
-## Highlighted Projects +## 精选项目 -Here we provide some impressive works you may find interesting: +这里提供一些你可能感兴趣的出色作品：
-| Title | Description | Links | +| 标题 | 描述 | 链接 | |:---:|:---:|:---:| -| [Semantic-SAM](https://github.com/UX-Decoder/Semantic-SAM) | A universal image segmentation model to enable segment and recognize anything at any desired granularity | [[Github](https://github.com/UX-Decoder/Semantic-SAM)]
[[Demo](https://github.com/UX-Decoder/Semantic-SAM)] | -| [SEEM: Segment Everything Everywhere All at Once](https://arxiv.org/pdf/2304.06718.pdf) | A powerful promptable segmentation model supports segmenting with various types of prompts (text, point, scribble, referring image, etc.) and any combination of prompts. | [[Github](https://github.com/UX-Decoder/Segment-Everything-Everywhere-All-At-Once)]
[[Demo](https://huggingface.co/spaces/xdecoder/SEEM)] | -| [OpenSeeD](https://arxiv.org/pdf/2303.08131.pdf) | A simple framework for open-vocabulary segmentation and detection which supports interactive segmentation with box input to generate mask | [[Github](https://github.com/IDEA-Research/OpenSeeD)] | -| [LLaVA](https://arxiv.org/abs/2304.08485) | Visual instruction tuning with GPT-4 | [[Github](https://github.com/haotian-liu/LLaVA)]
[[Page](https://llava-vl.github.io/)]
[[Demo](https://llava.hliu.cc/)]
[[Data](https://huggingface.co/datasets/liuhaotian/LLaVA-Instruct-150K)]
[[Model](https://huggingface.co/liuhaotian/LLaVA-13b-delta-v0)] | -| [GenSAM](https://arxiv.org/abs/2312.07374) | Relaxing the instance-specific manual prompt requirement in SAM through training-free test-time adaptation | [[Github](https://github.com/jyLin8100/GenSAM)]
[[Page](https://lwpyh.github.io/GenSAM/)] | +| [Semantic-SAM](https://github.com/UX-Decoder/Semantic-SAM) | 一个通用图像分割模型，可在任意所需粒度下分割并识别任意对象 | [[Github](https://github.com/UX-Decoder/Semantic-SAM)]
[[Demo](https://github.com/UX-Decoder/Semantic-SAM)] | +| [SEEM: Segment Everything Everywhere All at Once](https://arxiv.org/pdf/2304.06718.pdf) | 一个强大的可提示分割模型，支持使用多种类型的提示（文本、点、涂鸦、参考图像等）及其任意组合进行分割。 | [[Github](https://github.com/UX-Decoder/Segment-Everything-Everywhere-All-At-Once)]
[[Demo](https://huggingface.co/spaces/xdecoder/SEEM)] | +| [OpenSeeD](https://arxiv.org/pdf/2303.08131.pdf) | 一个用于开放词汇分割与检测的简单框架，支持通过框输入进行交互式分割以生成掩码 | [[Github](https://github.com/IDEA-Research/OpenSeeD)] | +| [LLaVA](https://arxiv.org/abs/2304.08485) | 基于 GPT-4 的视觉指令微调 | [[Github](https://github.com/haotian-liu/LLaVA)]
[[Page](https://llava-vl.github.io/)]
[[Demo](https://llava.hliu.cc/)]
[[Data](https://huggingface.co/datasets/liuhaotian/LLaVA-Instruct-150K)]
[[Model](https://huggingface.co/liuhaotian/LLaVA-13b-delta-v0)] | +| [GenSAM](https://arxiv.org/abs/2312.07374) | 通过无需训练的测试时自适应，放宽 SAM 中对实例特定手工提示的要求 | [[Github](https://github.com/jyLin8100/GenSAM)]
[[Page](https://lwpyh.github.io/GenSAM/)] |
-We also list some awesome segment-anything extension projects here you may find interesting: -- [Computer Vision in the Wild (CVinW) Readings](https://github.com/Computer-Vision-in-the-Wild/CVinW_Readings) for those who are interested in open-set tasks in computer vision. +我们还在此列出一些你可能感兴趣的出色 Segment Anything 扩展项目： +- [Computer Vision in the Wild (CVinW) Readings](https://github.com/Computer-Vision-in-the-Wild/CVinW_Readings) 适合对计算机视觉中开放集（open-set）任务感兴趣的用户。 - [Zero-Shot Anomaly Detection](https://github.com/caoyunkang/GroundedSAM-zero-shot-anomaly-detection) by Yunkang Cao - [EditAnything: ControlNet + StableDiffusion based on the SAM segmentation mask](https://github.com/sail-sg/EditAnything) by Shanghua Gao and Pan Zhou - [IEA: Image Editing Anything](https://github.com/feizc/IEA) by Zhengcong Fei @@ -145,12 +150,12 @@ We also list some awesome segment-anything extension projects here you may find - [DragGAN](https://github.com/Zeqiang-Lai/DragGAN) by Shanghai AI Lab. - [Tabletop HandyBot: Robotic arm assistant that performs tabletop tasks using Grounded-SAM](https://github.com/ycheng517/tabletop-handybot) by Yifei Cheng -## Installation -The code requires `python>=3.8`, as well as `pytorch>=1.7` and `torchvision>=0.8`. Please follow the instructions [here](https://pytorch.org/get-started/locally/) to install both PyTorch and TorchVision dependencies. Installing both PyTorch and TorchVision with CUDA support is strongly recommended. +## 安装 +代码需要 `python>=3.8`，以及 `pytorch>=1.7` 和 `torchvision>=0.8`。请按照[此处](https://pytorch.org/get-started/locally/) 的说明安装 PyTorch 和 TorchVision 依赖。强烈建议安装支持 CUDA 的 PyTorch 和 TorchVision。 -### Install with Docker +### 使用 Docker 安装 -Open one terminal: +打开一个终端： ``` make build-image @@ -160,50 +165,50 @@ make build-image make run ``` -That's it. +就这样。 -If you would like to allow visualization across docker container, open another terminal and type: +如果你希望跨 Docker 容器进行可视化，请打开另一个终端并输入： ``` xhost + ``` -### Install without Docker -You should set the environment variable manually as follows if you want to build a local GPU environment for Grounded-SAM: +### 不使用 Docker 安装 +如果你想为 Grounded-SAM 构建本地 GPU 环境，应按如下方式手动设置环境变量： ```bash export AM_I_DOCKER=False export BUILD_WITH_CUDA=True export CUDA_HOME=/path/to/cuda-11.3/ ``` -Install Segment Anything: +安装 Segment Anything： ```bash python -m pip install -e segment_anything ``` -Install Grounding DINO: +安装 Grounding DINO： ```bash pip install --no-build-isolation -e GroundingDINO ``` -Install diffusers: +安装 diffusers： ```bash pip install --upgrade diffusers[torch] ``` -Install osx: +安装 osx： ```bash git submodule update --init --recursive cd grounded-sam-osx && bash install.sh ``` -Install RAM & Tag2Text: +安装 RAM 与 Tag2Text： ```bash git clone https://github.com/xinyu1205/recognize-anything.git @@ -211,32 +216,32 @@ pip install -r ./recognize-anything/requirements.txt pip install -e ./recognize-anything/ ``` -The following optional dependencies are necessary for mask post-processing, saving masks in COCO format, the example notebooks, and exporting the model in ONNX format. `jupyter` is also required to run the example notebooks. +以下可选依赖项对于掩码后处理、以 COCO 格式保存掩码、示例 notebook 以及以 ONNX 格式导出模型是必需的。运行示例 notebook 还需要 `jupyter`。 ``` pip install opencv-python pycocotools matplotlib onnxruntime onnx ipykernel ``` -More details can be found in [install segment anything](https://github.com/facebookresearch/segment-anything#installation) and [install GroundingDINO](https://github.com/IDEA-Research/GroundingDINO#install) and [install OSX](https://github.com/IDEA-Research/OSX) +更多详情请参阅 [install segment anything](https://github.com/facebookresearch/segment-anything#installation) 和 [install GroundingDINO](https://github.com/IDEA-Research/GroundingDINO#install) 以及 [install OSX](https://github.com/IDEA-Research/OSX) ## Grounded-SAM Playground -Let's start exploring our Grounding-SAM Playground and we will release more interesting demos in the future, stay tuned! +让我们开始探索 Grounded-SAM Playground，未来我们还会发布更多有趣的演示，敬请期待！ -## :open_book: Step-by-Step Notebook Demo -Here we list some notebook demo provided in this project: +## :open_book: 分步 Notebook 演示 +此处列出本项目提供的一些 notebook 演示： - [grounded_sam.ipynb](grounded_sam.ipynb) - [grounded_sam_colab_demo.ipynb](grounded_sam_colab_demo.ipynb) - [grounded_sam_3d_box.ipynb](grounded_sam_3d_box) -### :running_man: GroundingDINO: Detect Everything with Text Prompt +### :running_man: GroundingDINO：用文本提示检测一切 :grapes: [[arXiv Paper](https://arxiv.org/abs/2303.05499)]   :rose:[[Try the Colab Demo](https://colab.research.google.com/github/roboflow-ai/notebooks/blob/main/notebooks/zero-shot-object-detection-with-grounding-dino.ipynb)]   :sunflower: [[Try Huggingface Demo](https://huggingface.co/spaces/ShilongLiu/Grounding_DINO_demo)]   :mushroom: [[Automated Dataset Annotation and Evaluation](https://youtu.be/C4NqaRBz_Kw)] -Here's the step-by-step tutorial on running `GroundingDINO` demo: +以下是运行 `GroundingDINO` 演示的分步教程： -**Step 1: Download the pretrained weights** +**步骤 1：下载预训练权重** ```bash cd Grounded-Segment-Anything @@ -245,14 +250,14 @@ cd Grounded-Segment-Anything wget https://github.com/IDEA-Research/GroundingDINO/releases/download/v0.1.0-alpha/groundingdino_swint_ogc.pth ``` -**Step 2: Running the demo** +**步骤 2：运行演示** ```bash python grounding_dino_demo.py ```
- Running with Python (same as demo but you can run it anywhere after installing GroundingDINO) + 使用 Python 运行（与演示相同，但在安装 GroundingDINO 后可在任意位置运行） ```python from groundingdino.util.inference import load_model, load_image, predict, annotate @@ -281,16 +286,16 @@ cv2.imwrite("annotated_image.jpg", annotated_frame)

-**Tips** -- If you want to detect multiple objects in one sentence with [Grounding DINO](https://github.com/IDEA-Research/GroundingDINO), we suggest separating each name with `.` . An example: `cat . dog . chair .` +**提示** +- 如果你想使用 [Grounding DINO](https://github.com/IDEA-Research/GroundingDINO), 在一句话中检测多个对象，我们建议用 `.` 分隔每个名称。示例：`cat . dog . chair .` -**Step 3: Check the annotated image** +**步骤 3：查看标注图像** -The annotated image will be saved as `./annotated_image.jpg`. +标注图像将保存为 `./annotated_image.jpg`。
-| Text Prompt | Demo Image | Annotated Image | +| 文本提示词 | 示例图像 | 标注图像 | |:----:|:----:|:----:| | `Bear.` | ![](./assets/demo1.jpg) | ![](./assets/annotated_image.jpg) | | `Horse. Clouds. Grasses. Sky. Hill` | ![](./assets/demo7.jpg) | ![](https://github.com/IDEA-Research/detrex-storage/blob/main/assets/grounded_sam/grounding_dino/groundingdino_demo7.jpg?raw=true) @@ -298,11 +303,11 @@ The annotated image will be saved as `./annotated_image.jpg`.
-### :running_man: Grounded-SAM: Detect and Segment Everything with Text Prompt +### :running_man: Grounded-SAM：使用文本提示词检测并分割一切 -Here's the step-by-step tutorial on running `Grounded-SAM` demo: +以下是运行 `Grounded-SAM` 演示的分步教程： -**Step 1: Download the pretrained weights** +**步骤 1：下载预训练权重** ```bash cd Grounded-Segment-Anything @@ -311,11 +316,11 @@ wget https://dl.fbaipublicfiles.com/segment_anything/sam_vit_h_4b8939.pth wget https://github.com/IDEA-Research/GroundingDINO/releases/download/v0.1.0-alpha/groundingdino_swint_ogc.pth ``` -We provide two versions of Grounded-SAM demo here: -- [grounded_sam_demo.py](./grounded_sam_demo.py): our original implementation for Grounded-SAM. -- [grounded_sam_simple_demo.py](./grounded_sam_simple_demo.py) our updated more elegant version for Grounded-SAM. +我们在此提供两个版本的 Grounded-SAM 演示： +- [grounded_sam_demo.py](./grounded_sam_demo.py)：Grounded-SAM 的原始实现。 +- [grounded_sam_simple_demo.py](./grounded_sam_simple_demo.py) 我们更新后的、更简洁的 Grounded-SAM 版本。 -**Step 2: Running original grounded-sam demo** +**步骤 2：运行原始 grounded-sam 演示** ```bash # depends on your device export CUDA_VISIBLE_DEVICES=0 @@ -335,25 +340,25 @@ python grounded_sam_demo.py \ --device "cuda" ``` -The annotated results will be saved in `./outputs` as follows +标注结果将按如下方式保存至 `./outputs`
-| Input Image | Annotated Image | Generated Mask | +| 输入图像 | 标注图像 | 生成的掩码 | |:----:|:----:|:----:| | ![](./assets/demo1.jpg) | ![](https://github.com/IDEA-Research/detrex-storage/blob/main/assets/grounded_sam/grounded_sam/original_grounded_sam_demo1.jpg?raw=true) | ![](https://github.com/IDEA-Research/detrex-storage/blob/main/assets/grounded_sam/grounded_sam/mask.jpg?raw=true) |
-**Step 3: Running grounded-sam demo with sam-hq** -- Download the demo image +**步骤 3：使用 sam-hq 运行 grounded-sam 演示** +- 下载演示图像 ```bash wget https://github.com/IDEA-Research/detrex-storage/releases/download/grounded-sam-storage/sam_hq_demo_image.png ``` -- Download SAM-HQ checkpoint [here](https://github.com/SysCV/sam-hq#model-checkpoints) +- 在此下载 SAM-HQ 检查点 [here](https://github.com/SysCV/sam-hq#model-checkpoints) -- Running grounded-sam-hq demo as follows: +- 按如下方式运行 grounded-sam-hq 演示： ```python export CUDA_VISIBLE_DEVICES=0 python grounded_sam_demo.py \ @@ -369,36 +374,36 @@ python grounded_sam_demo.py \ --device "cuda" ``` -The annotated results will be saved in `./outputs` as follows +标注结果将按如下方式保存至 `./outputs`
-| Input Image | SAM Output | SAM-HQ Output | +| 输入图像 | SAM 输出 | SAM-HQ 输出 | |:----:|:----:|:----:| | ![](https://github.com/IDEA-Research/detrex-storage/blob/main/assets/grounded_sam/sam_hq/sam_hq_demo.png?raw=true) | ![](https://github.com/IDEA-Research/detrex-storage/blob/main/assets/grounded_sam/sam_hq/sam_output.jpg?raw=true) | ![](https://github.com/IDEA-Research/detrex-storage/blob/main/assets/grounded_sam/sam_hq/sam_hq_output.jpg?raw=true) |
-**Step 4: Running the updated grounded-sam demo (optional)** +**步骤 4：运行更新版 grounded-sam 演示（可选）** -Note that this demo is almost same as the original demo, but **with more elegant code**. +请注意，此演示与原始演示几乎相同，但**代码更简洁**。 ```python python grounded_sam_simple_demo.py ``` -The annotated results will be saved as `./groundingdino_annotated_image.jpg` and `./grounded_sam_annotated_image.jpg` +标注结果将保存为 `./groundingdino_annotated_image.jpg` 和 `./grounded_sam_annotated_image.jpg`
-| Text Prompt | Input Image | GroundingDINO Annotated Image | Grounded-SAM Annotated Image | +| 文本提示词 | 输入图像 | GroundingDINO 标注图像 | Grounded-SAM 标注图像 | |:----:|:----:|:----:|:----:| | `The running dog` | ![](./assets/demo2.jpg) | ![](https://github.com/IDEA-Research/detrex-storage/blob/main/assets/grounded_sam/grounded_sam/groundingdino_annotated_image_demo2.jpg?raw=true) | ![](https://github.com/IDEA-Research/detrex-storage/blob/main/assets/grounded_sam/grounded_sam/grounded_sam_annotated_image_demo2.jpg?raw=true) | | `Horse. Clouds. Grasses. Sky. Hill` | ![](./assets/demo7.jpg) | ![](assets/groundingdino_annotated_image.jpg) | ![](assets/grounded_sam_annotated_image.jpg) |
-**Step 5: Running the Sam model with multi-gpu** +**步骤 5：在多 GPU 上运行 Sam 模型** ```bash export CUDA_VISIBLE_DEVICES=0,1 ``` @@ -415,11 +420,11 @@ python grounded_sam_multi_gpu_demo.py \ --text_prompt "car" \ --device "cuda" ``` -You will see that the model is loaded once per GPU ![](assets/multi-gpu.png) +你会看到每个 GPU 各加载一次模型 ![](assets/multi-gpu.png) -### :skier: Grounded-SAM with Inpainting: Detect, Segment and Generate Everything with Text Prompt +### :skier: 带修复（Inpainting）的 Grounded-SAM：使用文本提示词检测、分割并生成一切 -**Step 1: Download the pretrained weights** +**步骤 1：下载预训练权重** ```bash cd Grounded-Segment-Anything @@ -428,7 +433,7 @@ wget https://dl.fbaipublicfiles.com/segment_anything/sam_vit_h_4b8939.pth wget https://github.com/IDEA-Research/GroundingDINO/releases/download/v0.1.0-alpha/groundingdino_swint_ogc.pth ``` -**Step 2: Running grounded-sam inpainting demo** +**步骤 2：运行 grounded-sam 修复演示** ```bash CUDA_VISIBLE_DEVICES=0 @@ -445,51 +450,50 @@ python grounded_sam_inpainting_demo.py \ --device "cuda" ``` -The annotated and inpaint image will be saved in `./outputs` +标注图像和修复图像将保存至 `./outputs` -**Step 3: Check the results** +**步骤 3：查看结果**
-| Input Image | Det Prompt | Annotated Image | Inpaint Prompt | Inpaint Image | +| 输入图像 | 检测提示词 | 标注图像 | 修复提示词 | 修复图像 | |:---:|:---:|:---:|:---:|:---:| |![](./assets/inpaint_demo.jpg) | `Bench` | ![](https://github.com/IDEA-Research/detrex-storage/blob/main/assets/grounded_sam/grounded_sam_inpaint/grounded_sam_output.jpg?raw=true) | `A sofa, high quality, detailed` | ![](https://github.com/IDEA-Research/detrex-storage/blob/main/assets/grounded_sam/grounded_sam_inpaint/grounded_sam_inpainting_output.jpg?raw=true) |
-### :golfing: Grounded-SAM and Inpaint Gradio APP +### :golfing: Grounded-SAM 与修复 Gradio 应用 -We support 6 tasks in the local Gradio APP： +本地 Gradio 应用支持 6 种任务： -1. **scribble**: Segmentation is achieved through Segment Anything and mouse click interaction (you need to click on the object with the mouse, no need to specify the prompt). -2. **automask**: Segment the entire image at once through Segment Anything (no need to specify a prompt). -3. **det**: Realize detection through Grounding DINO and text interaction (text prompt needs to be specified). -4. **seg**: Realize text interaction by combining Grounding DINO and Segment Anything to realize detection + segmentation (need to specify text prompt). -5. **inpainting**: By combining Grounding DINO + Segment Anything + Stable Diffusion to achieve text exchange and replace the target object (need to specify text prompt and inpaint prompt) . -6. **automatic**: By combining BLIP + Grounding DINO + Segment Anything to achieve non-interactive detection + segmentation (no need to specify prompt). +1. **scribble**：通过 Segment Anything 与鼠标点击交互实现分割（需用鼠标点击目标对象，无需指定提示词）。 +2. **automask**：通过 Segment Anything 一次性分割整张图像（无需指定提示词）。 +3. **det**：通过 Grounding DINO 与文本交互实现检测（需指定文本提示词）。 +4. **seg**：结合 Grounding DINO 与 Segment Anything，通过文本交互实现检测 + 分割（需指定文本提示词）。 +5. **inpainting**：结合 Grounding DINO + Segment Anything + Stable Diffusion，实现文本替换并替换目标对象（需指定文本提示词和修复提示词）。 +6. **automatic**：结合 BLIP + Grounding DINO + Segment Anything，实现无需交互的检测 + 分割（无需指定提示词）。 ```bash python gradio_app.py ``` -- The gradio_app visualization as follows: +- gradio_app 可视化效果如下： ![](./assets/gradio_demo.png) -### :label: Grounded-SAM with RAM or Tag2Text for Automatic Labeling -[**The Recognize Anything Models**](https://github.com/OPPOMKLab/recognize-anything) are a series of open-source and strong fundamental image recognition models, including [RAM++](https://arxiv.org/abs/2310.15200), [RAM](https://arxiv.org/abs/2306.03514) and [Tag2text](https://arxiv.org/abs/2303.05657). +### :label: 结合 RAM 或 Tag2Text 的 Grounded-SAM 自动标注 +[**Recognize Anything Models（识别一切模型）**](https://github.com/OPPOMKLab/recognize-anything) 是一系列开源且强大的基础图像识别模型，包括 [RAM++](https://arxiv.org/abs/2310.15200), [RAM](https://arxiv.org/abs/2306.03514) 和 [Tag2text](https://arxiv.org/abs/2303.05657). -It is seamlessly linked to generate pseudo labels automatically as follows: -1. Use RAM/Tag2Text to generate tags. -2. Use Grounded-Segment-Anything to generate the boxes and masks. +可与 Grounded-SAM 无缝衔接，按如下步骤自动生成伪标签： +1. 使用 RAM/Tag2Text 生成标签。 +2. 使用 Grounded-Segment-Anything 生成边界框和掩码。 +**步骤 1：初始化子模块并下载预训练检查点** -**Step 1: Init submodule and download the pretrained checkpoint** - -- Init submodule: +- 初始化子模块： ```bash cd Grounded-Segment-Anything @@ -497,7 +501,7 @@ git submodule init git submodule update ``` -- Download pretrained weights for `GroundingDINO`, `SAM` and `RAM/Tag2Text`: +- 下载 `GroundingDINO`、`SAM` 和 `RAM/Tag2Text` 的预训练权重： ```bash wget https://dl.fbaipublicfiles.com/segment_anything/sam_vit_h_4b8939.pth @@ -508,7 +512,7 @@ wget https://huggingface.co/spaces/xinyu1205/Tag2Text/resolve/main/ram_swin_larg wget https://huggingface.co/spaces/xinyu1205/Tag2Text/resolve/main/tag2text_swin_14m.pth ``` -**Step 2: Running the demo with RAM** +**步骤 2：使用 RAM 运行演示** ```bash export CUDA_VISIBLE_DEVICES=0 python automatic_label_ram_demo.py \ @@ -525,7 +529,7 @@ python automatic_label_ram_demo.py \ ``` -**Step 2: Or Running the demo with Tag2Text** +**步骤 2：或使用 Tag2Text 运行演示** ```bash export CUDA_VISIBLE_DEVICES=0 python automatic_label_tag2text_demo.py \ @@ -541,20 +545,20 @@ python automatic_label_tag2text_demo.py \ --device "cuda" ``` -- RAM++ significantly improves the open-set capability of RAM, for [RAM++ inference on unseen categoreis](https://github.com/xinyu1205/recognize-anything#ram-inference-on-unseen-categories-open-set). -- Tag2Text also provides powerful captioning capabilities, and the process with captions can refer to [BLIP](#robot-run-grounded-segment-anything--blip-demo). -- The pseudo labels and model prediction visualization will be saved in `output_dir` as follows (right figure): +- RAM++ 显著提升了 RAM 的开放集（open-set）能力，关于[对未见类别进行 RAM++ 推理](https://github.com/xinyu1205/recognize-anything#ram-inference-on-unseen-categories-open-set). +- Tag2Text 同样提供强大的图像描述（captioning）能力，带描述文本的流程可参考 [BLIP](#robot-run-grounded-segment-anything--blip-demo)。 +- 伪标签与模型预测可视化结果将保存在 `output_dir`，如下所示（右图）： ![](./assets/automatic_label_output/demo9_tag2text_ram.jpg) -### :robot: Grounded-SAM with BLIP for Automatic Labeling -It is easy to generate pseudo labels automatically as follows: -1. Use BLIP (or other caption models) to generate a caption. -2. Extract tags from the caption. We use ChatGPT to handle the potential complicated sentences. -3. Use Grounded-Segment-Anything to generate the boxes and masks. +### :robot: 使用 BLIP 的 Grounded-SAM 自动标注 +可以按如下方式轻松自动生成伪标签： +1. 使用 BLIP（或其他描述模型）生成图像描述。 +2. 从描述中提取标签。我们使用 ChatGPT 处理可能较复杂的句子。 +3. 使用 Grounded-Segment-Anything 生成边界框和掩码。 -- Run Demo +- 运行 Demo ```bash export OPENAI_API_KEY=your_openai_key export OPENAI_API_BASE=https://closeai.deno.dev/v1 @@ -572,27 +576,27 @@ python automatic_label_demo.py \ --device "cuda" ``` -- When you don't have a paid Account for ChatGPT is also possible to use NLTK instead. Just don't include the ```openai_key``` Parameter when starting the Demo. - - The Script will automatically download the necessary NLTK Data. -- The pseudo labels and model prediction visualization will be saved in `output_dir` as follows: +- 如果你没有 ChatGPT 付费账号，也可以使用 NLTK。启动 Demo 时只需不包含 ```openai_key``` 参数即可。 + - 脚本会自动下载所需的 NLTK 数据。 +- 伪标签与模型预测可视化结果将保存在 `output_dir`，如下所示： ![](./assets/automatic_label_output_demo3.jpg) -### :open_mouth: Grounded-SAM with Whisper: Detect and Segment Anything with Audio -Detect and segment anything with speech! +### :open_mouth: 使用 Whisper 的 Grounded-SAM：用音频检测并分割任意目标 +用语音检测并分割任意目标！ ![](assets/acoustics/gsam_whisper_inpainting_demo.png) -**Install Whisper** +**安装 Whisper** ```bash pip install -U openai-whisper ``` -See the [whisper official page](https://github.com/openai/whisper#setup) if you have other questions for the installation. +若对安装有其他问题，请参阅 [Whisper 官方页面](https://github.com/openai/whisper#setup) -**Run Voice-to-Label Demo** +**运行语音转标签 Demo** -Optional: Download the demo audio file +可选：下载演示音频文件 ```bash wget https://huggingface.co/ShilongLiu/GroundingDINO/resolve/main/demo_audio.mp3 @@ -615,11 +619,11 @@ python grounded_sam_whisper_demo.py \ ![](./assets/grounded_sam_whisper_output.jpg) -**Run Voice-to-inpaint Demo** +**运行语音转修复（inpaint）Demo** -You can enable chatgpt to help you automatically detect the object and inpainting order with `--enable_chatgpt`. +你可以启用 ChatGPT，通过 `--enable_chatgpt` 自动检测目标及修复顺序。 -Or you can specify the object you want to inpaint [stored in `args.det_speech_file`] and the text you want to inpaint with [stored in `args.inpaint_speech_file`]. +或者，你可以指定要修复的目标 [保存在 `args.det_speech_file`] 以及用于修复的文本 [保存在 `args.inpaint_speech_file`]。 ```bash export OPENAI_API_KEY=your_openai_key @@ -658,22 +662,22 @@ python grounded_sam_whisper_inpainting_demo.py \ ![](./assets/acoustics/gsam_whisper_inpainting_pipeline.png) -### :speech_balloon: Grounded-SAM ChatBot Demo +### :speech_balloon: Grounded-SAM ChatBot 演示 https://user-images.githubusercontent.com/24236723/231955561-2ae4ec1a-c75f-4cc5-9b7b-517aa1432123.mp4 -Following [Visual ChatGPT](https://github.com/microsoft/visual-chatgpt), we add a ChatBot for our project. Currently, it supports: -1. "Describe the image." -2. "Detect the dog (and the cat) in the image." -3. "Segment anything in the image." -4. "Segment the dog (and the cat) in the image." -5. "Help me label the image." -6. "Replace the dog with a cat in the image." +参考 [Visual ChatGPT](https://github.com/microsoft/visual-chatgpt),，我们为项目添加了 ChatBot。目前支持： +1. "描述这张图像。" +2. "检测图像中的狗（和猫）。" +3. "分割图像中的任意目标。" +4. "分割图像中的狗（和猫）。" +5. "帮我标注这张图像。" +6. "将图像中的狗替换为猫。" -To use the ChatBot: -- Install whisper if you want to use audio as input. -- Set the default model setting in the tool `Grounded_dino_sam_inpainting`. -- Run Demo +使用 ChatBot： +- 若要以音频作为输入，请安装 Whisper。 +- 在工具 `Grounded_dino_sam_inpainting` 中设置默认模型配置。 +- 运行 Demo ```bash export OPENAI_API_KEY=your_openai_key export OPENAI_API_BASE=https://closeai.deno.dev/v1 @@ -681,7 +685,7 @@ export CUDA_VISIBLE_DEVICES=0 python chatbot.py ``` -### :man_dancing: Run Grounded-Segment-Anything + OSX Demo +### :man_dancing: 运行 Grounded-Segment-Anything + OSX 演示

@@ -689,10 +693,10 @@ python chatbot.py

-- Download the checkpoint `osx_l_wo_decoder.pth.tar` from [here](https://drive.google.com/drive/folders/1x7MZbB6eAlrq5PKC9MaeIm4GqkBpokow?usp=share_link) for OSX: -- Download the human model files and place it into `grounded-sam-osx/utils/human_model_files` following the instruction of [OSX](https://github.com/IDEA-Research/OSX). +- 从[此处](https://drive.google.com/drive/folders/1x7MZbB6eAlrq5PKC9MaeIm4GqkBpokow?usp=share_link) 下载 OSX 所需的检查点 `osx_l_wo_decoder.pth.tar`： +- 按照 [OSX](https://github.com/IDEA-Research/OSX). 的说明下载人体模型文件，并放入 `grounded-sam-osx/utils/human_model_files` -- Run Demo +- 运行 Demo ```shell export CUDA_VISIBLE_DEVICES=0 @@ -709,27 +713,27 @@ python grounded_sam_osx_demo.py \ --device "cuda" ``` -- The model prediction visualization will be saved in `output_dir` as follows: +- 模型预测可视化结果将保存在 `output_dir`，如下所示： -- We also support promptable 3D whole-body mesh recovery. For example, you can track someone with a text prompt and estimate his 3D pose and shape : +- 我们还支持可提示的 3D 全身网格恢复。例如，你可以用文本提示跟踪某人，并估计其 3D 姿态与体型： | ![space-1.jpg](assets/osx/grounded_sam_osx_output1.jpg) | | :---------------------------------------------------: | -| *A person with pink clothes* | +| *穿着粉色衣服的人* | | ![space-1.jpg](assets/osx/grounded_sam_osx_output2.jpg) | | :---------------------------------------------------: | -| *A man with a sunglasses* | +| *戴着太阳镜的男人* | -## :man_dancing: Run Grounded-Segment-Anything + VISAM Demo +## :man_dancing: 运行 Grounded-Segment-Anything + VISAM 演示 -- Download the checkpoint `motrv2_dancetrack.pth` from [here](https://drive.google.com/file/d/1EA4lndu2yQcVgBKR09KfMe5efbf631Th/view?usp=share_link) for MOTRv2: -- See the more thing if you have other questions for the installation. +- 从 [此处](https://drive.google.com/file/d/1EA4lndu2yQcVgBKR09KfMe5efbf631Th/view?usp=share_link) for MOTRv2: 下载 checkpoint `motrv2_dancetrack.pth` +- 若对安装还有其他问题，请参阅更多信息。 -- Run Demo +- 运行演示 ```shell export CUDA_VISIBLE_DEVICES=0 @@ -749,39 +753,39 @@ python grounded_sam_visam.py \ |![](https://raw.githubusercontent.com/BingfengYan/MOTSAM/main/visam.gif)| -### :dancers: Interactive Editing -- Release the interactive fashion-edit playground in [here](https://github.com/IDEA-Research/Grounded-Segment-Anything/tree/humanFace). Run in the notebook, just click for annotating points for further segmentation. Enjoy it! +### :dancers: 交互式编辑 +- 交互式时尚编辑 playground 已发布，见 [此处](https://github.com/IDEA-Research/Grounded-Segment-Anything/tree/humanFace). 在 notebook 中运行，只需点击即可标注点以进行进一步分割。尽情体验吧！ -- Release human-face-edit branch [here](https://github.com/IDEA-Research/Grounded-Segment-Anything/tree/humanFace). We'll keep updating this branch with more interesting features. Here are some examples: +- 人脸编辑分支已发布，见 [此处](https://github.com/IDEA-Research/Grounded-Segment-Anything/tree/humanFace). 我们将持续为该分支更新更多有趣功能。以下是一些示例： ![](https://github.com/IDEA-Research/Grounded-Segment-Anything/blob/humanFace/assets/231-hair-edit.png) -## :camera: 3D-Box via Segment Anything -We extend the scope to 3D world by combining Segment Anything and [VoxelNeXt](https://github.com/dvlab-research/VoxelNeXt). When we provide a prompt (e.g., a point / box), the result is not only 2D segmentation mask, but also 3D boxes. Please check [voxelnext_3d_box](./voxelnext_3d_box/) for more details. +## :camera: 基于 Segment Anything 的 3D 框 +我们通过结合 Segment Anything 与 [VoxelNeXt](https://github.com/dvlab-research/VoxelNeXt). 将应用范围扩展到 3D 世界。当我们提供提示（例如点 / 框）时，结果不仅是 2D 分割掩码（segmentation mask），还包括 3D 框。更多细节请查看 [voxelnext_3d_box](./voxelnext_3d_box/)。 ![](https://github.com/IDEA-Research/Grounded-Segment-Anything/blob/main/voxelnext_3d_box/images/sam-voxelnext.png) ![](https://github.com/IDEA-Research/Grounded-Segment-Anything/blob/main/voxelnext_3d_box/images/image_boxes2.png) -## :cupid: Acknowledgements +## :cupid: 致谢 - [Segment Anything](https://github.com/facebookresearch/segment-anything) - [Grounding DINO](https://github.com/IDEA-Research/GroundingDINO) -## Contributors +## 贡献者 -Our project wouldn't be possible without the contributions of these amazing people! Thank you all for making this project better. +没有这些优秀人士的贡献，我们的项目不可能实现！感谢大家让本项目变得更好。 -## Citation -If you find this project helpful for your research, please consider citing the following BibTeX entry. +## 引用 +若觉得本项目对你的研究有帮助，请考虑引用以下 BibTeX 条目。 ```BibTex @article{kirillov2023segany, title={Segment Anything},