wehub-resource-sync
263 lines
15 KiB
Markdown
263 lines
15 KiB
Markdown
<!-- WEHUB_ZH_README -->
|
||
> [!NOTE]
|
||
> 本文档由 WeHub 基于上游 README 翻译整理，属于社区翻译，非官方中文文档。
|
||
> [English](./README.en.md) · [原始项目](https://github.com/microsoft/JARVIS) · [上游 README](https://github.com/microsoft/JARVIS/blob/HEAD/README.md)
|
||
> 原作者、版权与许可证归属以原始项目及本仓库 LICENSE 文件为准。
|
||
|
||
# JARVIS
|
||
|
||
|
||
[](https://arxiv.org/abs/2303.17580)
|
||
[](https://huggingface.co/spaces/microsoft/HuggingGPT)
|
||
|
||
JARVIS 的使命是探索通用人工智能（AGI），并将前沿研究成果带给整个社区。
|
||
|
||
## 最新动态
|
||
|
||
+ [2024.01.15] 我们发布了 Easytool，以便更轻松地使用工具。
|
||
+ 代码和数据集可在 [EasyTool](/easytool) 获取。
|
||
+ 论文可在 [EasyTool: Enhancing LLM-based Agents with Concise Tool Instruction](https://arxiv.org/abs/2401.06201). 获取。
|
||
+ [2023.11.30] 我们发布了 TaskBench，用于评估大语言模型（LLM）的任务自动化能力。
|
||
+ 代码和数据集可在 [TaskBench](/taskbench) 获取。
|
||
+ 论文可在 [TaskBench: Benchmarking Large Language Models for Task Automation](https://arxiv.org/abs/2311.18760). 获取。
|
||
+ [2023.07.28] 我们正在进行评估规划和项目重建。我们将在不久的将来发布 Jarvis 的新版本。
|
||
+ [2023.07.24] 我们发布了 Jarvis 的轻量级 langchain 版本。见 <a href="https://github.com/langchain-ai/langchain/tree/master/libs/experimental/langchain_experimental/autonomous_agents/hugginggpt">此处</a>。
|
||
+ [2023.04.16] Jarvis 现已支持 Azure 平台上的 OpenAI 服务以及 GPT-4 模型。
|
||
+ [2023.04.06] 我们添加了 Gradio 演示，并在 `server` 模式下为 `/tasks` 和 `/results` 构建了 Web API。
|
||
+ Gradio 演示现已托管在 Hugging Face Space 上。（使用 `inference_mode=hybrid` 和 `local_deployment=standard` 构建）
|
||
+ Web API `/tasks` 和 `/results` 可访问 `Stage #1` 的中间结果：任务规划，以及 `Stage #1-3`：带执行结果的模型选择。见 <a href="#Server">此处</a>。
|
||
+ [2023.04.03] 我们添加了 CLI 模式，并提供了用于配置本地端点规模的参数。
|
||
+ 无需在本地部署模型，即可享受 Jarvis 的轻量体验。见 <a href="#Configuration">此处</a>。
|
||
+ 只需运行 `python awesome_chat.py --config configs/config.lite.yaml` 即可体验。
|
||
+ [2023.04.01] 我们更新了用于构建的代码版本。
|
||
|
||
|
||
### 概览
|
||
|
||
语言充当接口，使 LLM 能够连接众多 AI 模型以解决复杂的 AI 任务！
|
||
|
||
<p align="center">
|
||
<img width="100%" alt="image" src="./hugginggpt/assets/intro.png">
|
||
</p>
|
||
|
||
|
||
请参阅我们的论文：[HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in HuggingFace](http://arxiv.org/abs/2303.17580), Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu and Yueting Zhuang（前两位作者贡献相同）
|
||
|
||
我们介绍了一个协作系统，由 **LLM 作为控制器** 和 **众多专家模型作为协作执行者**（来自 HuggingFace Hub）组成。我们系统的工作流程包含四个阶段：
|
||
+ **任务规划（Task Planning）**：使用 ChatGPT 分析用户请求以理解其意图，并将其拆解为可能可解决的任务。
|
||
+ **模型选择（Model Selection）**：为解决规划好的任务，ChatGPT 根据专家模型在 Hugging Face 上的描述进行选择。
|
||
+ **任务执行（Task Execution）**：调用并执行每个所选模型，并将结果返回给 ChatGPT。
|
||
+ **响应生成（Response Generation）**：最后，使用 ChatGPT 整合所有模型的预测结果并生成响应。
|
||
|
||
<p align="center"><img src="./hugginggpt/assets/overview.jpg"></p>
|
||
|
||
### 系统要求
|
||
|
||
#### 默认（推荐）
|
||
|
||
适用于 `configs/config.default.yaml`：
|
||
|
||
+ Ubuntu 16.04 LTS
|
||
+ VRAM >= 24GB
|
||
+ RAM > 12GB（最低），16GB（标准），80GB（完整）
|
||
+ 磁盘 > 284GB
|
||
+ 42GB 用于 `damo-vilab/text-to-video-ms-1.7b`
|
||
+ 126GB 用于 `ControlNet`
|
||
+ 66GB 用于 `stable-diffusion-v1-5`
|
||
+ 50GB 用于其他
|
||
|
||
#### 最低（精简版）
|
||
|
||
适用于 `configs/config.lite.yaml`：
|
||
|
||
+ Ubuntu 16.04 LTS
|
||
+ 无其他要求
|
||
|
||
配置 `configs/config.lite.yaml` 无需在本地下载和部署任何专家模型。不过，这意味着 Jarvis 仅限于在 HuggingFace Inference Endpoints 上稳定运行的模型。
|
||
|
||
### 快速开始
|
||
|
||
请先在 `server/configs/config.default.yaml` 中将 `openai.key` 和 `huggingface.token` 替换为 **你的个人 OpenAI Key** 和 **你的 Hugging Face Token**，或分别将它们放入环境变量 `OPENAI_API_KEY` 和 `HUGGINGFACE_ACCESS_TOKEN`。然后运行以下命令：
|
||
|
||
<span id="Server"></span>
|
||
|
||
#### 服务端：
|
||
|
||
```bash
|
||
# setup env
|
||
cd server
|
||
conda create -n jarvis python=3.8
|
||
conda activate jarvis
|
||
conda install pytorch torchvision torchaudio pytorch-cuda=11.7 -c pytorch -c nvidia
|
||
pip install -r requirements.txt
|
||
|
||
# download models. Make sure that `git-lfs` is installed.
|
||
cd models
|
||
bash download.sh # required when `inference_mode` is `local` or `hybrid`.
|
||
|
||
# run server
|
||
cd ..
|
||
python models_server.py --config configs/config.default.yaml # required when `inference_mode` is `local` or `hybrid`
|
||
python awesome_chat.py --config configs/config.default.yaml --mode server # for text-davinci-003
|
||
```
|
||
|
||
现在你可以通过 Web API 访问 Jarvis 的服务。
|
||
|
||
+ `/hugginggpt` --method `POST`，访问完整服务。
|
||
+ `/tasks` --method `POST`，访问第 1 阶段的中间结果。
|
||
+ `/results` --method `POST`，访问第 1-3 阶段的中间结果。
|
||
|
||
例如：
|
||
|
||
```bash
|
||
# request
|
||
curl --location 'http://localhost:8004/tasks' \
|
||
--header 'Content-Type: application/json' \
|
||
--data '{
|
||
"messages": [
|
||
{
|
||
"role": "user",
|
||
"content": "based on pose of /examples/d.jpg and content of /examples/e.jpg, please show me a new image"
|
||
}
|
||
]
|
||
}'
|
||
|
||
# response
|
||
[{"args":{"image":"/examples/d.jpg"},"dep":[-1],"id":0,"task":"openpose-control"},{"args":{"image":"/examples/e.jpg"},"dep":[-1],"id":1,"task":"image-to-text"},{"args":{"image":"<GENERATED>-0","text":"<GENERATED>-1"},"dep":[1,0],"id":2,"task":"openpose-text-to-image"}]
|
||
```
|
||
|
||
|
||
#### Web 端：
|
||
|
||
我们提供了一个用户友好的网页。在以服务端模式启动 `awesome_chat.py` 后，你可以运行以下命令在浏览器中与 Jarvis 交互：
|
||
|
||
- 你需要先安装 `nodejs` 和 `npm`。
|
||
- [ 重要 ] 如果你在另一台机器上运行 Web 客户端，需要将 `http://{LAN_IP_of_the_server}:{port}/` 设置为 `web/src/config/index.ts` 的 `HUGGINGGPT_BASE_URL`。
|
||
- 如果你想使用视频生成功能，需要手动使用 H.264 编译 `ffmpeg`。
|
||
- 你可以通过设置图标上的 `double click` 切换到 ChatGPT！
|
||
|
||
```bash
|
||
cd web
|
||
npm install
|
||
npm run dev
|
||
```
|
||
|
||
```bash
|
||
# Optional: Install ffmpeg
|
||
# This command need be executed without errors.
|
||
LD_LIBRARY_PATH=/usr/local/lib /usr/local/bin/ffmpeg -i input.mp4 -vcodec libx264 output.mp4
|
||
```
|
||
|
||
<span id="Gradio"></span>
|
||
|
||
#### Gradio
|
||
|
||
Gradio 演示现已托管在 Hugging Face Space 上。你也可以运行以下命令在本地启动演示：
|
||
|
||
```bash
|
||
python models_server.py --config configs/config.gradio.yaml
|
||
python run_gradio_demo.py --config configs/config.gradio.yaml
|
||
|
||
# or run the HF Space as a Docker image (Build with `inference_mode=hibrid` and `local_deployment=standard`)
|
||
docker run -it -p 7860:7860 --platform=linux/amd64 registry.hf.space/microsoft-hugginggpt:latest python app.py
|
||
```
|
||
|
||
#### CLI：
|
||
|
||
你也可以在 CLI 模式下更轻松地运行 Jarvis：
|
||
|
||
```bash
|
||
cd server
|
||
python awesome_chat.py --config configs/config.default.yaml --mode cli
|
||
```
|
||
|
||
CLI 模式示例：
|
||
|
||
```
|
||
Welcome to Jarvis! A collaborative system that consists of an LLM as the controller and numerous expert models as collaborative executors. Jarvis can plan tasks, schedule Hugging Face models, generate friendly responses based on your requests, and help you with many things. Please enter your request (`exit` to exit).
|
||
[ User ]: Given a collection of image A: /examples/a.jpg, B: /examples/b.jpg, C: /examples/c.jpg, please tell me how many zebras in these picture?
|
||
[ Jarvis ]: Based on your request, I have used several models to process the images you provided. For the image A: /examples/a.jpg, I used the image-to-text model nlpconnect/vit-gpt2-image-captioning to generate the text "a cat sitting on a window sill looking out". Then I used the object-detection model facebook/detr-resnet-50 to generate the image with predicted box '/images/a3dc.jpg', which contains two objects with labels 'potted plant' and 'cat'. For the image B: /examples/b.jpg, I used the image-to-text model nlpconnect/vit-gpt2-image-captioning to generate the text "two zebras standing in a field of dry grass". Then I used the object-detection model facebook/detr-resnet-50 to generate the image with predicted box '/images/f5df.jpg', which contains three objects with labels 'zebra'. For the image C: /examples/c.jpg, I used the image-to-text model nlpconnect/vit-gpt2-image-captioning to generate the text "a zebra standing in a field of tall grass". Then I used the object-detection model facebook/detr-resnet-50 to generate the image with predicted box '/images/69c7.jpg', which contains one object with label 'zebra'. Finally, I used the canny-control model canny-control to generate the image '/images/7e18.png' based on the image /examples/f.jpg. Therefore, there are four zebras in these pictures. Is there anything else I can help you with?
|
||
[ User ]: Please answer all the named entities in the sentence: Iron Man is a superhero appearing in American comic books published by Marvel Comics. The character was co-created by writer and editor Stan Lee, developed by scripter Larry Lieber, and designed by artists Don Heck and Jack Kirby.
|
||
[ Jarvis ]: Sure, I understand your request. Based on the inference results of the models, I will answer your request step by step. Firstly, I used the token-classification model dslim/bert-base-NER to recognize the named entities in the sentence. The inference result is that the named entities are Iron Man, Stan Lee, Larry Lieber, Don Heck and Jack Kirby. Then, I used the question-answering model deepset/roberta-base-squad2 to answer your request. The inference result is that there is no output since the context cannot be empty. Therefore, I cannot make it. I hope this answer your request. Is there anything else I can help you with?
|
||
[ User ]: Please generate a canny image based on /examples/f.jpg
|
||
[ Jarvis ]: Sure. I have generated a canny image based on /examples/f.jpg. To do this, I first used the image-to-text model nlpconnect/vit-gpt2-image-captioning to generate the text description of the image, which is "a herd of giraffes and zebras grazing in a field". Then I used the canny-control model to generate the canny image of the original image, which is stored in the path "/images/1ea2.png". Finally, I used the canny-text-to-image model lllyasviel/sd-controlnet-canny to generate the canny image based on the text description and the canny image, which is stored in the path "/images/ba66.png". Do you need anything else?
|
||
```
|
||
|
||
### 配置
|
||
|
||
服务端配置文件为 `server/configs/config.default.yaml`，部分参数如下所示：
|
||
|
||
+ `model`：LLM，当前支持 `text-davinci-003`。我们正在努力集成更多开源 LLM。
|
||
+ `inference_mode`：推理端点（inference endpoints）模式
|
||
+ `local`：仅使用本地推理端点
|
||
+ `huggingface`：仅使用 Hugging Face Inference Endpoints **（无需本地推理端点）**
|
||
+ `hybrid`：同时使用 `local` 和 `huggingface`
|
||
+ `local_deployment`：本地部署模型的规模，适用于 `local` 或 `hybrid` 推理模式：
|
||
+ `minimal` (RAM>12GB, ControlNet only)
|
||
+ `standard` (RAM>16GB, ControlNet + Standard Pipelines)
|
||
+ `full` (RAM>42GB, All registered models)
|
||
|
||
在个人笔记本电脑上，我们推荐使用 `inference_mode: hybrid ` 和 `local_deployment: minimal` 的配置。但由于远程 Hugging Face Inference Endpoints 的不稳定性，在此设置下可用模型可能会受到限制。
|
||
|
||
### NVIDIA Jetson 嵌入式设备支持
|
||
项目包含一个 [Dockerfile](./Dockerfile.jetson)，为 [NVIDIA Jetson 嵌入式设备](https://developer.nvidia.com/embedded-computing). 该镜像提供加速版 ffmpeg、pytorch、torchaudio 和 torchvision 依赖。要构建 Docker 镜像，请[确保默认 Docker 运行时设置为 'nvidia'](https://github.com/NVIDIA/nvidia-docker/wiki/Advanced-topics#default-runtime). 预构建镜像见 https://hub.docker.com/r/toolboc/nv-jarvis.
|
||
|
||
```bash
|
||
#Build the docker image
|
||
docker build --pull --rm -f "Dockerfile.jetson" -t toolboc/nv-jarvis:r35.2.1
|
||
```
|
||
|
||
由于内存要求，JARVIS 需要在 Jetson AGX Orin 系列设备上运行（优先选择板载 64G RAM 的设备），并将配置选项设置为：
|
||
* `inference_mode: local`
|
||
* `local_deployment: standard`
|
||
|
||
建议通过从宿主机到容器的卷挂载提供模型和配置，如下方 `docker run` 步骤所示。你也可以取消注释 [Dockerfile](./Dockerfile.jetson) 中的 `# Download local models` 部分，以构建包含模型的容器。
|
||
|
||
#### 在 Jetson Orin AGX 上启动模型服务器、awesomechat 和 Web 应用
|
||
|
||
```bash
|
||
# run the container which will automatically start the model server
|
||
docker run --name jarvis --net=host --gpus all -v ~/jarvis/configs:/app/server/configs -v ~/src/JARVIS/server/models:/app/server/models toolboc/nv-jarvis:r35.2.1
|
||
|
||
# (wait for model server to complete initialization)
|
||
|
||
# start awesome_chat.py
|
||
docker exec jarvis python3 awesome_chat.py --config configs/config.default.yaml --mode server
|
||
|
||
#start the web application (application will be acessible at http://localhost:9999)
|
||
docker exec jarvis npm run dev --prefix=/app/web
|
||
```
|
||
|
||
### 截图
|
||
|
||
<p align="center"><img src="./hugginggpt/assets/screenshot_q.jpg"><img src="./hugginggpt/assets/screenshot_a.jpg"></p>
|
||
|
||
|
||
|
||
|
||
## Citation
|
||
如果你在方法中觉得本工作有用，可以按如下方式引用论文：
|
||
|
||
@inproceedings{shen2023hugginggpt,
|
||
author = {Shen, Yongliang and Song, Kaitao and Tan, Xu and Li, Dongsheng and Lu, Weiming and Zhuang, Yueting},
|
||
booktitle = {Advances in Neural Information Processing Systems},
|
||
title = {HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in HuggingFace},
|
||
year = {2023}
|
||
}
|
||
|
||
>
|
||
@article{shen2023taskbench,
|
||
title = {TaskBench: Benchmarking Large Language Models for Task Automation},
|
||
author = {Shen, Yongliang and Song, Kaitao and Tan, Xu and Zhang, Wenqi and Ren, Kan and Yuan, Siyu and Lu, Weiming and Li, Dongsheng and Zhuang, Yueting},
|
||
journal = {arXiv preprint arXiv:2311.18760},
|
||
year = {2023}
|
||
}
|
||
|
||
>
|
||
@article{yuan2024easytool,
|
||
title = {EASYTOOL: Enhancing LLM-based Agents with Concise Tool Instruction},
|
||
author = {Siyu Yuan and Kaitao Song and Jiangjie Chen and Xu Tan and Yongliang Shen and Ren Kan and Dongsheng Li and Deqing Yang},
|
||
journal = {arXiv preprint arXiv:2401.06201},
|
||
year = {2024}
|
||
}
|