chore: import upstream snapshot with attribution
This commit is contained in:
wehub-resource-sync
@@ -0,0 +1,360 @@
|
||||
<p align="center">
|
||||
<img src="./assets/logo2.png" width="20%"> <br>
|
||||
</p>
|
||||
|
||||
<div align="center">
|
||||
<h1>TaskBench</h1>
|
||||
<div align="center">
|
||||
<a href="https://opensource.org/licenses/Apache-2.0">
|
||||
<img alt="License: Apache 2.0" src="https://img.shields.io/badge/License-Apache%202.0-4E94CE.svg">
|
||||
</a>
|
||||
<a href="https://arxiv.org/abs/2311.18760">
|
||||
<img alt="License: Apache 2.0" src="https://img.shields.io/badge/arXiv-Paper-<COLOR>.svg">
|
||||
</a>
|
||||
</div>
|
||||
<h3>Benchmarking Large Language Models for Task Automation<h3>
|
||||
</div>
|
||||
|
||||
<p align="center">
|
||||
<img width="100%" alt="image" src="./assets/eval.png">
|
||||
</p>
|
||||
|
||||
|
||||
## What's New
|
||||
|
||||
+ [2023.11.30] We release TaskBench for evaluating the task automation capability of LLMs.
|
||||
+ The code and datasets are available at [TaskBench](#).
|
||||
+ The paper is available at [TaskBench: Benchmarking Large Language Models for Task Automation](https://arxiv.org/abs/2311.18760).
|
||||
|
||||
## Table of Contents
|
||||
|
||||
+ [Introduction](#introduction)
|
||||
+ [Dataset](#dataset)
|
||||
+ [Evaluation with TaskBench](#evaluation-with-taskbench)
|
||||
+ [Dataset Construction with Back-Instruct](#dataset-construction-with-back-instruct)
|
||||
+ [Leaderboard](#leaderboard)
|
||||
+ [Citation](#citation)
|
||||
|
||||
## Introduction
|
||||
|
||||
TaskBench is a benchmark for evaluating large language models (LLMs) on task automation. Task automation can be formulated into three critical stages: task decomposition, tool invocation, and parameter prediction. This complexity makes data collection and evaluation more challenging compared to common NLP tasks. To address this challenge, we propose a comprehensive evaluation framework and a high-quality dataset for task automation. We also provide a leaderboard of 17 LLMs on TaskBench, including GPT-4, Claude-2, and other open-source LLMs.
|
||||
|
||||
|
||||
### Dataset
|
||||
|
||||
To generate high-quality evaluation datasets, we introduce the concept of Tool Graph to represent the decomposed tasks in user intent, and adopt a Back-Instruct method to simulate user instruction and annotations. The data collection process consists of three stages:
|
||||
|
||||
+ **Tool Graph Construction:** we first build a tool library and use the tool library to construct the tool graph. The nodes in the tool graph represent the tools, and the edges represent the dependencies between the tools, including the resource dependency and temporal dependency.
|
||||
+ **Graph Sampling:** we sample the tool graph to generate the tool graph for each sample. The sampled tool graph is used to generate the tool invocation graph and the instruction. According to the topology of the sampled tool graph, we sample the tool graph in three ways: node, chain and DAGs, which represent different structures of task decomposition for task automation.
|
||||
+ **Back-Instruct:** we first use the sampled tool graph to generate the task steps and the instructions. Then, we use the instruction to generate the tool invocation parameters to complete the tool invocation graph.
|
||||
|
||||
<p align="center">
|
||||
<img width="100%" alt="image" src="./assets/backinstruct.png">
|
||||
</p>
|
||||
|
||||
To improve the quality of the dataset, we use LLM-based and rule-based critics to verify the dataset. The former aims to use LLM to check the alignments between the generated data and the sampled tool graph. While the latter uses straightforward rules to determine the alignment between the tool graphs in created data and the sampled tool graphs. Here, we use the nodes and edges of the sampled graph to determine the consistency. Details statistics of the processing are shown in [the table](#LLM-based-and-Rule-based-Critics).
|
||||
|
||||
After LLM-based and rule-based critics, we further verify the dataset with human annotators, including checking the syntax of the instructions, the correctness of the tool invocation graph, and the correctness of the tool invocation parameters. The final dataset contains 28,271 samples in three domains: HuggingFace Tools, Multimedia Tools, and Daily Life APIs. Details statistics of the human verification are shown in [the table](#Human-Verification).
|
||||
|
||||
#### Introduction
|
||||
|
||||
The TaskBench dataset contains datasets in three areas: HuggingFace Tools, Multimedia Tools, and Dailylife APIs. Each dataset directory includes three files:
|
||||
|
||||
+ `data.json`: the dataset file, which contains the samples in the dataset.
|
||||
+ `graph_desc.json`: the tool graph description file, which contains the tool graph of the dataset.
|
||||
+ `user_requests.json`: contains the user requests of the dataset.
|
||||
+ `tool_desc.json`: the tool description file, which contains the tool descriptions of the dataset.
|
||||
|
||||
```
|
||||
├─data_dailylifeapis
|
||||
│ data.json
|
||||
│ graph_desc.json
|
||||
│ user_requests.json
|
||||
│ tool_desc.json
|
||||
│
|
||||
├─data_huggingface
|
||||
│ data.json
|
||||
│ graph_desc.json
|
||||
│ user_requests.json
|
||||
│ tool_desc.json
|
||||
│
|
||||
└─data_multimedia
|
||||
data.json
|
||||
graph_desc.json
|
||||
user_requests.json
|
||||
tool_desc.json
|
||||
```
|
||||
|
||||
#### Processing Statistics
|
||||
|
||||
We provide the statistics of the dataset processing in the following tables:
|
||||
|
||||
+ **Overview**: we provide the number of samples in each dataset, the number of samples checked by critics, and the number of samples verified by humans. Grouped by the tool invocation graph structure, e.g. node, chain, and DAGs, we also provide the number of samples in each group.
|
||||
+ **LLM-based and Rule-based Critics**: we provide the number of samples checked by LLM-based critics, rule-based critics and both critics.
|
||||
+ **Human Verification**: Human verification is built on the samples checked by critics, which includes three parts: syntax checking, instruction checking, and tool invocation graph checking. We provide the number of samples in each part, and along with the number of samples that are discarded or fixed.
|
||||
|
||||
| Dataset | #Samples | #Samples Checked by Critics (%) | #Samples Verified by Humans (%) | Node | Chain | DAG |
|
||||
| :-----: | :------: | :----------------: | :--------------: | :------: | :------: | :------: |
|
||||
| Hugging Face Models | 12,217 | 8,457 (69.22%) | 7,546 (61.76%) | 3,067 | 3,642 | 837 |
|
||||
| Multimedia Tools | 8,904 | 6,281 (70.54%) | 5,584 (62.71%) | 2,037 | 2,982 | 565 |
|
||||
| Dailylife APIs | 7,150 | 5,432 (75.97%) | 4,320 (60.42%) | 1,258 | 2,787 | 275 |
|
||||
|
||||
<div id="LLM-based-and-Rule-based-Critics">
|
||||
|
||||
| Dataset | #Samples | #Checked by LLM-based Critics (%) | #Checked by Rule-based Critics (%) | #Checked by Both Critics (%) |
|
||||
| :-----: | :------: | :-----------------------------: | :------------------------------: | :-------------------------: |
|
||||
| Hugging Face Models | 12,217 | 9,042 (74.01%) | 10,289 (84.22%) | 8,457 (69.22%) |
|
||||
| Multimedia Tools | 8,904 | 6,959 (78.16%) | 7,363 (82.69%) | 6,281 (70.54%) |
|
||||
| Dailylife APIs | 7,150 | 5,694 (79.63%) | 6,271 (87.70%) | 5,432 (75.97%) |
|
||||
|
||||
<div id="Human-Verification">
|
||||
|
||||
| Dataset | #Samples Checked by Critics | #Correct Samples (%) | #Discarded (%) | #Fixed for Syntax (%) | #Fixed for Instructions (%) | #Fixed for Tool Invocation Graph (%) |
|
||||
| :-----: | :-------------------------: | :-------------------: | :-------------------: | :---------------------------: | :-----------------------------------: | :------------: |
|
||||
| Hugging Face Models | 8,457 | 6,974 (82.46%) | 911 (10.77%) | 27 (0.32%) | 328 (3.87%) | 843 (9.96%) |
|
||||
| Multimedia Tools | 6,281 | 5,262 (83.77%) | 697 (11.09%) | 11 (0.17%) | 107 (1.70%) | 526 (9.96%) |
|
||||
| Dailylife APIs | 5,432 | 4,307 (79.29%) | 714 (13.14%) | 6 (0.11%) | 92 (1.68%) | 332 (6.11%) |
|
||||
|
||||
## Evaluation with TaskBench
|
||||
|
||||
On top of the TaskBench dataset, we provide a comprehensive evaluation framework for task automation. The evaluation framework consists of three stages: task decomposition, tool invocation, and parameter prediction. We provide the evaluation metrics for each stage:
|
||||
|
||||
+ **Task Decomposition**: Since task steps are diverse text distributions, we use the Rouge-1 (R1), Rouge-2 (R2), and Bertscore F1 (BsF) metrics to evaluate the task decomposition results.
|
||||
+ **Tool Invocation**: We report the F1 of node prediction (n-F1) and edge prediction (e-F1) in the tool invocation graph to evaluate the tool invocation results. Edge prediction reflects the correctness of the dependencies between tools, while node prediction reflects the correctness of the tool prediction.
|
||||
+ **Parameter Prediction**: For tool parameters prediction, we report the parameter type (or name) F1 (t-F1) and parameter value F1 (v-F1).
|
||||
|
||||
To evaluate the task automation performance of LLMs on TaskBench we provide the evaluation code and data, please follow the instructions below:
|
||||
|
||||
### Setup
|
||||
|
||||
```bash
|
||||
conda create -n taskbench python=3.8
|
||||
conda activate taskbench
|
||||
pip install -r requirements.txt
|
||||
```
|
||||
|
||||
Additionally, if you wish to evaluate open-source large language models, you will also need to deploy the LLMs locally using an **OpenAI-compatible API**. We recommend using the `fastchat` tool to deploy the service to the `localhost:4000` endpoint.
|
||||
|
||||
```bash
|
||||
pip install fastchat
|
||||
pip install vllm
|
||||
pip install "fastapi[all]"
|
||||
|
||||
python3 -m fastchat.serve.controller
|
||||
python3 -m fastchat.serve.vllm_worker --model-path lmsys/vicuna-7b-v1.3
|
||||
python3 -m fastchat.serve.openai_api_server --host localhost --port 4000
|
||||
```
|
||||
|
||||
### Inference
|
||||
|
||||
For convenience, it is recommended to deploy all LLMs to the same endpoint, such as `localhost:4000`. To generate the prediction file on TaskBench, specify the name of the LLM using the following command:
|
||||
|
||||
```bash
|
||||
export YOUR_API_KEY=API_KEY
|
||||
python inference.py \
|
||||
--llm gpt-4 \
|
||||
--data_dir data_multimedia \
|
||||
--temperature 0.2 \
|
||||
--top_p 0.1 \
|
||||
--api_addr localhost \
|
||||
--api_port 4000 \
|
||||
--api_key $YOUR_API_KEY \
|
||||
--multiworker 5 \
|
||||
--use_demos 0 \
|
||||
--reformat true \
|
||||
--reformat_by self \
|
||||
--log_first_detail true \
|
||||
--use_demos 2 \
|
||||
--dependency_type resource \
|
||||
--tag true
|
||||
```
|
||||
|
||||
### Evaluation
|
||||
|
||||
With the predictions in place, you can now evaluate the LLMs. The predictions file is saved by default in the dataset's folder under the name `predictions`. Execute the following command to calculate the evaluation metrics (saved in the `metrics` folder):
|
||||
|
||||
```bash
|
||||
python evaluate.py \
|
||||
--data_dir data_multimedia \
|
||||
--prediction_dir $prediction_dir \
|
||||
--llm gpt-4 \
|
||||
--splits all \
|
||||
--n_tools all \
|
||||
--mode add \
|
||||
--dependency_type resource \
|
||||
-m all
|
||||
```
|
||||
|
||||
## Dataset Construction with Back-Instruct
|
||||
|
||||
We have provided the dataset for three domains: Hugging Face Tools (`data_huggingface`), Multimedia Tools (`data_multimedia`), and Daily Life APIs (`data_dailylifeapis`). If you want to generate your own dataset, please follow the instructions below:
|
||||
|
||||
### Construct Your Own Tool Graph
|
||||
|
||||
First, you need to build your own tool library. The tool library is a JSON file that contains the description of the tools and tool parameters. Two formats of the tool are supported:
|
||||
|
||||
```json
|
||||
// Tool with type-specific parameters
|
||||
{
|
||||
"id": "Image-to-Image",
|
||||
"desc": "Image-to-image is the task of transforming a source image to match the characteristics of a target image or a target image domain. Any image manipulation and enhancement is possible with image to image models.",
|
||||
"input-type": [
|
||||
"image"
|
||||
],
|
||||
"output-type": [
|
||||
"image"
|
||||
]
|
||||
}
|
||||
// API with request parameters
|
||||
{
|
||||
"id": "send_sms",
|
||||
"desc": "Send an sms to a specific phone number",
|
||||
"parameters": [
|
||||
{
|
||||
"name": "phone_number",
|
||||
"type": "string",
|
||||
"desc": "The phone number to send the sms to"
|
||||
},
|
||||
{
|
||||
"name": "content",
|
||||
"type": "string",
|
||||
"desc": "The content of the sms"
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
Then based on the tool library, you can use the script `generate_graph.py` to generate the tool graph. Now we support two type of tool graph: resource dependency graph and temporal dependency graph. For type-specific parameters, we use the resource dependency graph. For API with request parameters, we use the temporal dependency graph. You can specify the tool graph type by the parameter `--dependency_type`. In the future, we will support more types of tool graphs.
|
||||
|
||||
```bash
|
||||
python generate_graph.py \
|
||||
--tool_desc tool_desc.json \
|
||||
--dependency_type resource \
|
||||
--data_dir data_multimedia
|
||||
```
|
||||
|
||||
> Note: The auto-generated tool graph may not be perfect. You can manually modify the tool graph to make it more reasonable. You can check the tool graph through the visualization tool `visualize_graph.py`. We recommend that you manually create the tool graph thoroughly, which will help you to generate a high-quality dataset.
|
||||
|
||||
### Generate the Dataset
|
||||
|
||||
After generating the tool graph, you can use the script `data_engine.py` to generate the dataset. You need to specify the tool graph description file to `--graph_desc` and the tool description file to `--tool_desc`.
|
||||
|
||||
```bash
|
||||
# specify the graph and tool description file
|
||||
python data_engine.py \
|
||||
--graph_desc data_multimedia/graph_desc.json \
|
||||
--tool_desc data_multimedia/tool_desc.json \
|
||||
--llm gpt-4 \
|
||||
--temperature 1.0 \
|
||||
--top_p 1.0 \
|
||||
--dependency_type resource \
|
||||
--save_figure false \
|
||||
--api_addr localhost \
|
||||
--api_port 4000 \
|
||||
--api_key $YOUR_API_KEY \
|
||||
--check true \
|
||||
--use_async true \
|
||||
--multiworker 5
|
||||
|
||||
# specify the data_dir to resume data generation
|
||||
python data_engine.py \
|
||||
--data_dir result_20240317170826_gpt-4-32k_t1_0_p1_0_check \
|
||||
--llm gpt-4-32k \
|
||||
--temperature 1.0 \
|
||||
--top_p 1.0 \
|
||||
--dependency_type temporal \
|
||||
--save_figure false \
|
||||
--api_addr localhost \
|
||||
--api_port 4000 \
|
||||
--api_key $YOUR_API_KEY \
|
||||
--check true \
|
||||
--use_async true \
|
||||
--multiworker 5
|
||||
|
||||
python format_data.py \
|
||||
--data_dir data_multimedia \
|
||||
--dependency_type resource
|
||||
```
|
||||
|
||||
## Leaderboard
|
||||
|
||||
Based on the evaluation framework and the TaskBench dataset, we provide a leaderboard of task automation performance of 17 LLMs. We provide the evaluation results of each LLM in the following tables:
|
||||
|
||||
### Multimedia Tools Domain
|
||||
|
||||
| LLM | R1 | R2 | BsF | n-F1 | e-F1 | t-F1 | v-F1 |
|
||||
|----------------------|-------|-------|------|------|------|------|------|
|
||||
| gpt-4 | 60.84 | 40.08 | 91.19 | 90.90 | 69.27 | 87.06 | 72.31 |
|
||||
| claude-2 | 48.85 | 23.59 | 89.22 | 80.94 | 53.01 | 71.63 | 51.58 |
|
||||
| gpt-3.5-turbo | 49.66 | 28.51 | 89.54 | 72.83 | 44.02 | 65.91 | 40.80 |
|
||||
| text-davinci-003 | 49.23 | 27.97 | 89.21 | 73.97 | 45.81 | 68.48 | 40.70 |
|
||||
| codellama-13b | 44.46 | 23.30 | 88.66 | 62.78 | 24.61 | 48.19 | 29.13 |
|
||||
| codellama-7b | 43.76 | 22.93 | 88.81 | 53.29 | 14.76 | 38.04 | 24.45 |
|
||||
| vicuna-13b-v1.5 | 44.75 | 23.75 | 88.94 | 60.61 | 14.78 | 41.62 | 23.62 |
|
||||
| nous-hermes-13b | 35.73 | 16.11 | 87.53 | 58.97 | 8.90 | 43.60 | 21.69 |
|
||||
| wizardlm-13b | 35.87 | 17.55 | 87.29 | 51.24 | 4.82 | 39.10 | 18.74 |
|
||||
| vicuna-7b-v1.5 | 39.46 | 19.83 | 88.53 | 46.06 | 4.26 | 29.72 | 13.74 |
|
||||
| longchat-7b-v1.5 | 37.85 | 18.14 | 87.64 | 43.08 | 3.95 | 27.89 | 13.41 |
|
||||
| baichuan-13b-chat | 20.41 | 3.77 | 83.31 | 42.51 | 5.19 | 28.04 | 11.77 |
|
||||
| llama-2-13b-chat | 26.16 | 7.88 | 84.82 | 43.87 | 1.63 | 29.99 | 11.32 |
|
||||
| internlm-chat-7b | 16.64 | 3.56 | 82.91 | 23.60 | 1.14 | 13.75 | 6.09 |
|
||||
| llama-2-7b-chat | 34.51 | 15.91 | 87.56 | 26.47 | 0.91 | 18.27 | 5.84 |
|
||||
| mpt-7b-chat | 30.94 | 11.90 | 86.08 | 8.68 | 0.18 | 3.19 | 1.02 |
|
||||
| vicuna-33b-v1.3 | 31.27 | 13.37 | 86.17 | 6.40 | 0.01 | 2.47 | 1.09 |
|
||||
|
||||
### HuggingFace Tools Domain
|
||||
|
||||
| LLM | R1 | R2 | BsF | n-F1 | e-F1 | t-F1 | v-F1 |
|
||||
|----------------------|-------|-------|------|------|------|------|------|
|
||||
| gpt-4 | 52.42 | 30.38 | 90.12 | 81.54 | 54.70 | 77.31 | 60.86 |
|
||||
| claude-2 | 44.21 | 21.12 | 88.71 | 79.00 | 43.51 | 63.00 | 43.08 |
|
||||
| text-davinci-003 | 36.68 | 17.61 | 87.03 | 59.38 | 29.37 | 52.53 | 36.04 |
|
||||
| gpt-3.5-turbo | 42.99 | 21.58 | 88.47 | 69.49 | 33.36 | 55.88 | 36.32 |
|
||||
| codellama-13b | 38.75 | 18.37 | 88.32 | 53.16 | 14.64 | 32.06 | 18.87 |
|
||||
| nous-hermes-13b | 37.36 | 16.91 | 88.18 | 53.62 | 8.29 | 37.51 | 17.66 |
|
||||
| wizardlm-13b | 34.47 | 15.38 | 87.38 | 54.40 | 2.05 | 38.76 | 15.35 |
|
||||
| llama-2-13b-chat | 39.37 | 18.64 | 88.67 | 48.47 | 7.30 | 31.61 | 15.38 |
|
||||
| longchat-7b-v1.5 | 27.09 | 8.97 | 85.50 | 48.18 | 0.56 | 33.57 | 13.94 |
|
||||
| baichuan-13b-chat | 19.93 | 5.97 | 83.85 | 53.85 | 7.65 | 33.17 | 13.53 |
|
||||
| vicuna-13b-v1.5 | 37.12 | 17.03 | 87.90 | 50.82 | 7.28 | 28.34 | 11.85 |
|
||||
| vicuna-7b-v1.5 | 27.17 | 10.02 | 85.61 | 42.87 | 2.76 | 24.65 | 10.81 |
|
||||
| vicuna-33b-v1.3 | 33.52 | 14.75 | 86.73 | 43.40 | 4.82 | 22.71 | 10.07 |
|
||||
| codellama-7b | 38.97 | 18.62 | 88.46 | 37.59 | 5.35 | 22.50 | 9.20 |
|
||||
| internlm-chat-7b | 20.53 | 7.16 | 83.74 | 24.39 | 0.83 | 15.41 | 6.64 |
|
||||
| llama-2-7b-chat | 24.12 | 8.68 | 85.43 | 27.30 | 0.74 | 13.05 | 2.79 |
|
||||
| mpt-7b-chat | 33.21 | 12.73 | 87.23 | 20.86 | 0.12 | 9.61 | 1.83 |
|
||||
|
||||
### Daily Life APIs Domain
|
||||
|
||||
| LLM | R1 | R2 | BsF | n-F1 | e-F1 | t-F1 | v-F1 |
|
||||
|----------------------|-------|-------|------|------|------|------|------|
|
||||
| gpt-4 | 85.07 | 72.36 | 96.91 | 96.91 | 80.53 | 97.02 | 71.14 |
|
||||
| claude-2 | 82.26 | 69.88 | 96.64 | 93.52 | 75.31 | 92.71 | 64.72 |
|
||||
| codellama-13b | 89.86 | 83.27 | 97.90 | 87.73 | 63.16 | 84.26 | 62.38 |
|
||||
| gpt-3.5-turbo | 58.53 | 39.90 | 91.29 | 85.37 | 60.67 | 81.97 | 55.66 |
|
||||
| text-davinci-003 | 68.27 | 50.30 | 93.59 | 80.42 | 54.90 | 78.37 | 53.40 |
|
||||
| nous-hermes-13b | 78.49 | 68.04 | 95.61 | 73.45 | 3.50 | 64.47 | 47.22 |
|
||||
| vicuna-13b-v1.5 | 81.76 | 71.76 | 96.31 | 75.67 | 12.48 | 64.27 | 47.31 |
|
||||
| wizardlm-13b | 82.02 | 72.43 | 96.36 | 69.34 | 14.18 | 55.00 | 40.53 |
|
||||
| codellama-7b | 56.98 | 38.83 | 91.31 | 59.33 | 27.23 | 52.99 | 34.81 |
|
||||
| vicuna-33b-v1.3 | 54.96 | 39.71 | 91.40 | 52.49 | 16.37 | 39.95 | 29.64 |
|
||||
| vicuna-7b-v1.5 | 40.26 | 21.19 | 87.27 | 52.73 | 14.23 | 36.30 | 24.67 |
|
||||
| baichuan-13b-chat | 49.43 | 27.25 | 88.32 | 52.55 | 10.61 | 37.48 | 23.77 |
|
||||
| llama-2-13b-chat | 45.39 | 22.42 | 87.74 | 55.77 | 17.02 | 35.11 | 22.94 |
|
||||
| longchat-7b-v1.5 | 29.05 | 14.84 | 83.90 | 47.26 | 14.44 | 25.73 | 18.18 |
|
||||
| internlm-chat-7b | 42.94 | 21.02 | 86.14 | 29.14 | 6.63 | 19.21 | 13.48 |
|
||||
| llama-2-7b-chat | 37.06 | 16.49 | 86.31 | 30.17 | 4.27 | 14.94 | 9.34 |
|
||||
| mpt-7b-chat | 44.54 | 20.98 | 87.17 | 15.95 | 1.69 | 5.34 | 3.45 |
|
||||
|
||||
More details can be found in our paper: [TaskBench: Benchmarking Large Language Models for Task Automation](https://arxiv.org/abs/2311.18760).
|
||||
|
||||
## Citation
|
||||
|
||||
If you find this work useful in your method, you can cite the paper as below:
|
||||
|
||||
@article{shen2023taskbench,
|
||||
title = {TaskBench: Benchmarking Large Language Models for Task Automation},
|
||||
author = {Shen, Yongliang and Song, Kaitao and Tan, Xu and Zhang, Wenqi and Ren, Kan and Yuan, Siyu and Lu, Weiming and Li, Dongsheng and Zhuang, Yueting},
|
||||
journal = {arXiv preprint arXiv:2311.18760},
|
||||
year = {2023}
|
||||
}
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 404 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 496 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 992 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 692 KiB |
@@ -0,0 +1,27 @@
|
||||
#!/bin/bash
|
||||
set -x
|
||||
set -e
|
||||
|
||||
export CUDA_VISIBLE_DEVICES=0
|
||||
|
||||
data_dir=$1
|
||||
prediction_dir=$2
|
||||
|
||||
if [[ $data_dir == *"dailylifeapis"* ]]; then
|
||||
dependency_type="temporal"
|
||||
else
|
||||
dependency_type="resource"
|
||||
fi
|
||||
|
||||
for file in $data_dir/$prediction_dir/*.json
|
||||
do
|
||||
llm=$(basename $file .json)
|
||||
# replace prediction_dir's "predictions" with "metrics"
|
||||
metrics=$(echo $prediction_dir | sed 's/predictions/metrics/g')
|
||||
if [ -f $data_dir/$metrics/${llm}_splits_all_tools_all_metric_all.json ] && [ -s $data_dir/$metrics/${llm}_splits_all_tools_all_metric_all.json ];
|
||||
then
|
||||
continue
|
||||
fi
|
||||
echo $llm
|
||||
python evaluate.py --data_dir $data_dir --prediction_dir $prediction_dir --llm $llm --splits all --n_tools all --mode add --dependency_type $dependency_type -m all
|
||||
done
|
||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because one or more lines are too long
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,565 @@
|
||||
{
|
||||
"nodes": [
|
||||
{
|
||||
"id": "get_weather",
|
||||
"desc": "Get the weather for a specific city and a specific day",
|
||||
"parameters": [
|
||||
{
|
||||
"name": "location",
|
||||
"type": "string",
|
||||
"desc": "The location to get the weather for"
|
||||
},
|
||||
{
|
||||
"name": "date",
|
||||
"type": "date",
|
||||
"desc": "The date to get the weather for"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "get_news_for_topic",
|
||||
"desc": "Get the news for a specific topic",
|
||||
"parameters": [
|
||||
{
|
||||
"name": "topic",
|
||||
"type": "string",
|
||||
"desc": "The topic to get the news for"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "stock_operation",
|
||||
"desc": "Do a specific operation on a specific stock",
|
||||
"parameters": [
|
||||
{
|
||||
"name": "stock",
|
||||
"type": "string",
|
||||
"desc": "The stock to do the operation on"
|
||||
},
|
||||
{
|
||||
"name": "operation",
|
||||
"type": "string",
|
||||
"desc": "The operation to do, eg. buy, sell, short, cover etc."
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "book_flight",
|
||||
"desc": "Book a flight for a specific date, from a specific location to a specific destination",
|
||||
"parameters": [
|
||||
{
|
||||
"name": "date",
|
||||
"type": "date",
|
||||
"desc": "The date to book the flight for"
|
||||
},
|
||||
{
|
||||
"name": "from",
|
||||
"type": "string",
|
||||
"desc": "The location to book the flight from"
|
||||
},
|
||||
{
|
||||
"name": "to",
|
||||
"type": "string",
|
||||
"desc": "The location to book the flight to"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "book_hotel",
|
||||
"desc": "Book a specific hotel for a specific date",
|
||||
"parameters": [
|
||||
{
|
||||
"name": "date",
|
||||
"type": "date",
|
||||
"desc": "The date to book the hotel for"
|
||||
},
|
||||
{
|
||||
"name": "name",
|
||||
"type": "string",
|
||||
"desc": "The name of the hotel to book"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "book_restaurant",
|
||||
"desc": "Book a specific restaurant for a specific date",
|
||||
"parameters": [
|
||||
{
|
||||
"name": "date",
|
||||
"type": "date",
|
||||
"desc": "The date to book the restaurant for"
|
||||
},
|
||||
{
|
||||
"name": "name",
|
||||
"type": "string",
|
||||
"desc": "The name of the restaurant to book"
|
||||
}
|
||||
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "book_car",
|
||||
"desc": "Book a car for a specific date, in a specific location",
|
||||
"parameters": [
|
||||
{
|
||||
"name": "date",
|
||||
"type": "date",
|
||||
"desc": "The date to book the car for"
|
||||
},
|
||||
{
|
||||
"name": "location",
|
||||
"type": "string",
|
||||
"desc": "The location to book the car in"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "online_shopping",
|
||||
"desc": "Buy a product from a specific website",
|
||||
"parameters": [
|
||||
{
|
||||
"name": "website",
|
||||
"type": "string",
|
||||
"desc": "The website to buy the product from, eg. Amazon, Ebay, Taobao etc."
|
||||
},
|
||||
{
|
||||
"name": "product",
|
||||
"type": "string",
|
||||
"desc": "The product to buy"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "send_email",
|
||||
"desc": "Send an email to a specific email address",
|
||||
"parameters": [
|
||||
{
|
||||
"name": "email_address",
|
||||
"type": "string",
|
||||
"desc": "The email address to send the email to"
|
||||
},
|
||||
{
|
||||
"name": "content",
|
||||
"type": "string",
|
||||
"desc": "The content of the email"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "send_sms",
|
||||
"desc": "Send an sms to a specific phone number",
|
||||
"parameters": [
|
||||
{
|
||||
"name": "phone_number",
|
||||
"type": "string",
|
||||
"desc": "The phone number to send the sms to"
|
||||
},
|
||||
{
|
||||
"name": "content",
|
||||
"type": "string",
|
||||
"desc": "The content of the sms"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "share_by_social_network",
|
||||
"desc": "Share a specific content by a specific social network",
|
||||
"parameters": [
|
||||
{
|
||||
"name": "content",
|
||||
"type": "string",
|
||||
"desc": "The content to share"
|
||||
},
|
||||
{
|
||||
"name": "social_network",
|
||||
"type": "string",
|
||||
"desc": "The social network to share the content by, eg. Wechat, Facebook, Twitter, Weibo etc."
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "search_by_engine",
|
||||
"desc": "Search a specific query by a specific search engine",
|
||||
"parameters": [
|
||||
{
|
||||
"name": "query",
|
||||
"type": "string",
|
||||
"desc": "The content to search"
|
||||
},
|
||||
{
|
||||
"name": "engine",
|
||||
"type": "string",
|
||||
"desc": "The search engine to use, eg. Google, Bing, Baidu etc."
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "apply_for_job",
|
||||
"desc": "Apply for a specific job",
|
||||
"parameters": [
|
||||
{
|
||||
"name": "job",
|
||||
"type": "string",
|
||||
"desc": "The job to apply for"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "see_doctor_online",
|
||||
"desc": "See a specific doctor for a specific disease",
|
||||
"parameters": [
|
||||
{
|
||||
"name": "disease",
|
||||
"type": "string",
|
||||
"desc": "The disease to see the doctor for"
|
||||
},
|
||||
{
|
||||
"name": "doctor",
|
||||
"type": "string",
|
||||
"desc": "The doctor to see"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "consult_lawyer_online",
|
||||
"desc": "Consult a specific lawyer for a specific legal issue",
|
||||
"parameters": [
|
||||
{
|
||||
"name": "issue",
|
||||
"type": "string",
|
||||
"desc": "The legal issue to consult the lawyer for"
|
||||
},
|
||||
{
|
||||
"name": "lawyer",
|
||||
"type": "string",
|
||||
"desc": "The lawyer to consult"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "enroll_in_course",
|
||||
"desc": "Enroll in a specific course at a specific university",
|
||||
"parameters": [
|
||||
{
|
||||
"name": "course",
|
||||
"type": "string",
|
||||
"desc": "The course to enroll in"
|
||||
},
|
||||
{
|
||||
"name": "university",
|
||||
"type": "string",
|
||||
"desc": "The university to enroll in the course at"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "buy_insurance",
|
||||
"desc": "Buy a specific insurance from a specific insurance company",
|
||||
"parameters": [
|
||||
{
|
||||
"name": "insurance",
|
||||
"type": "string",
|
||||
"desc": "The insurance to buy"
|
||||
},
|
||||
{
|
||||
"name": "company",
|
||||
"type": "string",
|
||||
"desc": "The insurance company to buy the insurance from"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "online_banking",
|
||||
"desc": "Do a specific banking operation online at a specific bank",
|
||||
"parameters": [
|
||||
{
|
||||
"name": "instruction",
|
||||
"type": "string",
|
||||
"desc": "The banking instruction to do, eg. transfer, deposit, withdraw etc."
|
||||
},
|
||||
{
|
||||
"name": "bank",
|
||||
"type": "string",
|
||||
"desc": "The bank to do the banking operation at"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "daily_bill_payment",
|
||||
"desc": "Pay a specific bill",
|
||||
"parameters": [
|
||||
{
|
||||
"name": "bill",
|
||||
"type": "string",
|
||||
"desc": "The bill to pay, eg. electricity, water, gas, phone, internet etc."
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "sell_item_online",
|
||||
"desc": "Sell a specific item at a specific online store",
|
||||
"parameters": [
|
||||
{
|
||||
"name": "item",
|
||||
"type": "string",
|
||||
"desc": "The item to sell"
|
||||
},
|
||||
{
|
||||
"name": "store",
|
||||
"type": "string",
|
||||
"desc": "The online store to sell the item at, eg. Amazon, Ebay, Taobao etc."
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "do_tax_return",
|
||||
"desc": "Do the tax return for a specific year",
|
||||
"parameters": [
|
||||
{
|
||||
"name": "year",
|
||||
"type": "string",
|
||||
"desc": "The year to do the tax return for"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "apply_for_passport",
|
||||
"desc": "Apply for a passport",
|
||||
"parameters": [
|
||||
{
|
||||
"name": "country",
|
||||
"type": "string",
|
||||
"desc": "The country to apply for the passport for"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "pay_for_credit_card",
|
||||
"desc": "Pay for a specific credit card",
|
||||
"parameters": [
|
||||
{
|
||||
"name": "credit_card",
|
||||
"type": "string",
|
||||
"desc": "The credit card to pay for"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "auto_housework_by_robot",
|
||||
"desc": "Let a robot do a housework by following a specific instruction",
|
||||
"parameters": [
|
||||
{
|
||||
"name": "instruction",
|
||||
"type": "string",
|
||||
"desc": "The instruction to let the robot follow, eg. clean the floor, wash the dishes, do the laundry etc."
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "auto_driving_to_destination",
|
||||
"desc": "Let a car drive to a specific destination",
|
||||
"parameters": [
|
||||
{
|
||||
"name": "destination",
|
||||
"type": "string",
|
||||
"desc": "The destination to drive to"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "deliver_package",
|
||||
"desc": "Deliver a specific package to a specific destination",
|
||||
"parameters": [
|
||||
{
|
||||
"name": "package",
|
||||
"type": "string",
|
||||
"desc": "The package to deliver"
|
||||
},
|
||||
{
|
||||
"name": "destination",
|
||||
"type": "string",
|
||||
"desc": "The destination to deliver the package to"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "order_food_delivery",
|
||||
"desc": "Order a specific food to be delivered to a specific location at a specific platform",
|
||||
"parameters": [
|
||||
{
|
||||
"name": "food",
|
||||
"type": "string",
|
||||
"desc": "The food to order"
|
||||
},
|
||||
{
|
||||
"name": "location",
|
||||
"type": "string",
|
||||
"desc": "The location to deliver the food to"
|
||||
},
|
||||
{
|
||||
"name": "platform",
|
||||
"type": "string",
|
||||
"desc": "The platform to order the food at, eg. Uber Eats, Meituan Waimai etc."
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "order_taxi",
|
||||
"desc": "Order a taxi to a specific location at a specific platform",
|
||||
"parameters": [
|
||||
{
|
||||
"name": "location",
|
||||
"type": "string",
|
||||
"desc": "The location to order the taxi to"
|
||||
},
|
||||
{
|
||||
"name": "platform",
|
||||
"type": "string",
|
||||
"desc": "The platform to order the taxi at, eg. Uber, Didi etc."
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "play_music_by_title",
|
||||
"desc": "Play a specific music by a specific title",
|
||||
"parameters": [
|
||||
{
|
||||
"name": "title",
|
||||
"type": "string",
|
||||
"desc": "The title of the music to play"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "play_movie_by_title",
|
||||
"desc": "Play a specific movie by a specific title",
|
||||
"parameters": [
|
||||
{
|
||||
"name": "title",
|
||||
"type": "string",
|
||||
"desc": "The title of the movie to play"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "take_note",
|
||||
"desc": "Take a note",
|
||||
"parameters": [
|
||||
{
|
||||
"name": "content",
|
||||
"type": "string",
|
||||
"desc": "The content of the note"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "borrow_book_online",
|
||||
"desc": "Borrow a specific book from a specific library",
|
||||
"parameters": [
|
||||
{
|
||||
"name": "book",
|
||||
"type": "string",
|
||||
"desc": "The book to borrow"
|
||||
},
|
||||
{
|
||||
"name": "library",
|
||||
"type": "string",
|
||||
"desc": "The library to borrow the book from"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "recording_audio",
|
||||
"desc": "Record an audio",
|
||||
"parameters": [
|
||||
{
|
||||
"name": "content",
|
||||
"type": "string",
|
||||
"desc": "The content of the audio"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "make_video_call",
|
||||
"desc": "Make a video call to a specific phone number",
|
||||
"parameters": [
|
||||
{
|
||||
"name": "phone_number",
|
||||
"type": "string",
|
||||
"desc": "The phone number to make the video call to"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "make_voice_call",
|
||||
"desc": "Make a voice call to a specific phone number",
|
||||
"parameters": [
|
||||
{
|
||||
"name": "phone_number",
|
||||
"type": "string",
|
||||
"desc": "The phone number to make the voice call to"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "organize_meeting_online",
|
||||
"desc": "Organize a meeting online about a specific topic",
|
||||
"parameters": [
|
||||
{
|
||||
"name": "topic",
|
||||
"type": "string",
|
||||
"desc": "The topic of the meeting"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "attend_meeting_online",
|
||||
"desc": "Attend a meeting online about a specific topic",
|
||||
"parameters": [
|
||||
{
|
||||
"name": "topic",
|
||||
"type": "string",
|
||||
"desc": "The topic of the meeting"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "software_management",
|
||||
"desc": "Manage a specific software by a specific instruction",
|
||||
"parameters": [
|
||||
{
|
||||
"name": "software",
|
||||
"type": "string",
|
||||
"desc": "The software to manage"
|
||||
},
|
||||
{
|
||||
"name": "instruction",
|
||||
"type": "string",
|
||||
"desc": "The instruction to manage the software by, eg. install, uninstall, update etc."
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "print_document",
|
||||
"desc": "Print a specific document",
|
||||
"parameters": [
|
||||
{
|
||||
"name": "document",
|
||||
"type": "string",
|
||||
"desc": "The document to print"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "set_alarm",
|
||||
"desc": "Set an alarm for a specific time",
|
||||
"parameters": [
|
||||
{
|
||||
"name": "time",
|
||||
"type": "string",
|
||||
"desc": "The time to set the alarm for"
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,355 @@
|
||||
import os
|
||||
import random
|
||||
import traceback
|
||||
import uuid
|
||||
import json
|
||||
import networkx as nx
|
||||
from datetime import datetime
|
||||
from concurrent.futures import ThreadPoolExecutor, as_completed
|
||||
import shutil
|
||||
import warnings
|
||||
|
||||
from graph_sampler import GraphSampler
|
||||
import click
|
||||
import matplotlib.pyplot as plt
|
||||
import asyncio
|
||||
import aiohttp
|
||||
import logging
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
logger.setLevel(logging.INFO)
|
||||
logger.handlers = []
|
||||
|
||||
console_handler = logging.StreamHandler()
|
||||
console_handler.setFormatter(logging.Formatter("%(asctime)s - %(name)s - %(levelname)s - %(message)s"))
|
||||
console_handler.setLevel(logging.INFO)
|
||||
logger.addHandler(console_handler)
|
||||
|
||||
@click.command()
|
||||
@click.option("--temperature", type=float, default=0.7)
|
||||
@click.option("--top_p", type=float, default=1)
|
||||
@click.option("--check", type=bool, default=False)
|
||||
@click.option("--data_dir", type=str, default=None)
|
||||
@click.option("--graph_desc", type=str, default=None)
|
||||
@click.option("--tool_desc", type=str, default=None)
|
||||
@click.option("--api_addr", type=str, default="localhost")
|
||||
@click.option("--api_port", type=int, default=4000)
|
||||
@click.option("--api_key", type=str, default="your api key")
|
||||
@click.option("--play", type=bool, default=False)
|
||||
@click.option("--method", type=str, default=None)
|
||||
@click.option("--tool_number", type=int, default=None)
|
||||
@click.option("--number_of_samples", type=int, default=5000)
|
||||
@click.option("--seed", type=int, default=-1)
|
||||
@click.option("--save_figure", type=bool, default=False)
|
||||
@click.option("--multiworker", type=int, default=1)
|
||||
@click.option("--llm", type=str, default="gpt-4")
|
||||
@click.option("--use_async", type=bool, default=False)
|
||||
@click.option("--dependency_type", type=str, default="resource")
|
||||
def main(temperature, top_p, check, graph_desc, tool_desc, api_addr, api_port, api_key, play, method, tool_number, number_of_samples, seed, data_dir, save_figure, multiworker, llm, use_async, dependency_type):
|
||||
args = locals()
|
||||
url = f"http://{api_addr}:{api_port}/v1/chat/completions"
|
||||
header = {
|
||||
"Authorization": f"Bearer {api_key}",
|
||||
"Content-Type": "application/json"
|
||||
}
|
||||
now = datetime.now()
|
||||
|
||||
if data_dir:
|
||||
if os.path.exists(data_dir):
|
||||
if graph_desc or tool_desc:
|
||||
warnings.warn(f"Data directory {data_dir} already exists, tool graph and tool desc will be ignored.")
|
||||
graph_desc = f"{data_dir}/graph_desc.json"
|
||||
tool_desc = f"{data_dir}/tool_desc.json"
|
||||
else:
|
||||
data_dir = f"result_{now.strftime('%Y%m%d%H%M%S')}_{llm}_t{temperature}_p{top_p}{'_check' if check else ''}".replace(".", "_")
|
||||
|
||||
assert data_dir and graph_desc and tool_desc
|
||||
|
||||
if seed > -1:
|
||||
random.seed(seed)
|
||||
|
||||
tool_list = json.load(open(tool_desc, "r"))["nodes"]
|
||||
tools = {}
|
||||
if dependency_type == "resource":
|
||||
assert "input-type" in tool_list[0] and "output-type" in tool_list[0], "Input and output types are not defined"
|
||||
for tool in tool_list:
|
||||
tools[tool["id"]] = {"id": tool["id"], "desc": tool["desc"], "input-type": tool["input-type"], "output-type": tool["output-type"]}
|
||||
elif dependency_type == "temporal":
|
||||
assert "parameters" in tool_list[0], "Parameters are not defined"
|
||||
for tool in tool_list:
|
||||
tools[tool["id"]] = {"id": tool["id"], "desc": tool["desc"], "parameters": tool["parameters"]}
|
||||
else:
|
||||
raise ValueError(f"Unsupported dependency type: {dependency_type}")
|
||||
|
||||
sampler = GraphSampler(file_name=graph_desc)
|
||||
|
||||
if play:
|
||||
assert method is not None
|
||||
assert tool_number is not None
|
||||
result = asyncio.run(sample(url, header, llm, temperature, top_p, check, tool_number, sampler, tools, method, "./", None, dependency_type))
|
||||
logger.info(json.dumps(result, indent=2))
|
||||
return
|
||||
|
||||
figure_dir = None
|
||||
|
||||
if not os.path.exists(data_dir):
|
||||
os.makedirs(data_dir, exist_ok=True)
|
||||
shutil.copy(graph_desc, f"{data_dir}/graph_desc.json")
|
||||
shutil.copy(tool_desc, f"{data_dir}/tool_desc.json")
|
||||
|
||||
output = f"{data_dir}/data_raw.json"
|
||||
statistics_output = f"{data_dir}/statistics.json"
|
||||
|
||||
file_handler = logging.FileHandler(f"{data_dir}/data_engine.log")
|
||||
file_handler.setFormatter(logging.Formatter("%(asctime)s - %(name)s - %(levelname)s - %(message)s"))
|
||||
file_handler.setLevel(logging.INFO)
|
||||
logger.addHandler(file_handler)
|
||||
|
||||
logger.info(json.dumps(args))
|
||||
|
||||
if save_figure:
|
||||
figure_dir = f"{data_dir}/task_graphs"
|
||||
os.makedirs(figure_dir, exist_ok=True)
|
||||
|
||||
wf = open(output, "a")
|
||||
statistics_wf = open(f"{statistics_output}", "a")
|
||||
args["start_time"] = now.strftime("%Y-%m-%d %H:%M:%S")
|
||||
statistics_wf.write(json.dumps(args) + "\n")
|
||||
|
||||
method_weights = {
|
||||
"single": 3,
|
||||
"chain": 7,
|
||||
"dag": 8,
|
||||
}
|
||||
|
||||
number_weights = {
|
||||
2: 0.1,
|
||||
3: 0.2,
|
||||
4: 0.3,
|
||||
5: 0.2,
|
||||
6: 0.1,
|
||||
7: 0.05,
|
||||
8: 0.025,
|
||||
9: 0.025,
|
||||
10: 0.025,
|
||||
}
|
||||
|
||||
statistics = {"total": 0, "avg_time_per_sample": 0, "success": 0, "fail": 0}
|
||||
done, failed = [], []
|
||||
if use_async:
|
||||
# coroutine with Semaphore
|
||||
sem = asyncio.Semaphore(multiworker)
|
||||
async def sample_with_statistics(url, header, llm, temperature, top_p, check, tool_number, sampler, tools, method, figure_dir, wf, statistics, now, dependency_type):
|
||||
async with sem: # semaphore limits num of simultaneous sampling
|
||||
if statistics["total"] % 100 == 0 and statistics["total"] != 0:
|
||||
logger.info(json.dumps(statistics, indent=2))
|
||||
statistics_wf.write(json.dumps(statistics) + "\n")
|
||||
try:
|
||||
await sample(url, header, llm, temperature, top_p, check, tool_number, sampler, tools, method, figure_dir, wf, dependency_type)
|
||||
except Exception as e:
|
||||
statistics["total"] += 1
|
||||
statistics["fail"] += 1
|
||||
if str(type(e)) not in statistics:
|
||||
statistics[str(type(e))] = 0
|
||||
statistics[str(type(e))] += 1
|
||||
raise e
|
||||
statistics["total"] += 1
|
||||
statistics["success"] += 1
|
||||
statistics["avg_time_per_sample"] = str((datetime.now() - now) / statistics["success"])
|
||||
|
||||
async def run(url, header, llm, temperature, top_p, check, sampler, tools, figure_dir, wf, statistics, now, dependency_type):
|
||||
method = random.choices(list(method_weights.keys()), weights=list(method_weights.values()))[0]
|
||||
if method == "single":
|
||||
tool_number = 1
|
||||
else:
|
||||
tool_number = random.choices(list(number_weights.keys()), weights=list(number_weights.values()))[0]
|
||||
if method == "dag":
|
||||
tool_number = max(tool_number, 3)
|
||||
await sample_with_statistics(url, header, llm, temperature, top_p, check, tool_number, sampler, tools, method, figure_dir, wf, statistics, now, dependency_type)
|
||||
|
||||
tasks = []
|
||||
for _ in range(number_of_samples):
|
||||
tasks.append(run(url, header, llm, temperature, top_p, check, sampler, tools, figure_dir, wf, statistics, now, dependency_type))
|
||||
|
||||
loop = asyncio.get_event_loop()
|
||||
results = loop.run_until_complete(asyncio.gather(*tasks, return_exceptions=True))
|
||||
|
||||
for result in results:
|
||||
if isinstance(result, Exception):
|
||||
failed.append(result)
|
||||
else:
|
||||
done.append(result)
|
||||
else:
|
||||
# multi-thread with ThreadPoolExecutor
|
||||
executor = ThreadPoolExecutor(max_workers=multiworker)
|
||||
def sample_with_statistics(url, header, llm, temperature, top_p, check, tool_number, sampler, tools, method, figure_dir, wf, statistics, now, dependency_type):
|
||||
if statistics["total"] % 100 == 0 and statistics["total"] != 0:
|
||||
logger.info(json.dumps(statistics, indent=2))
|
||||
statistics_wf.write(json.dumps(statistics) + "\n")
|
||||
try:
|
||||
asyncio.run(sample(url, header, llm, temperature, top_p, check, tool_number, sampler, tools, method, figure_dir, wf, dependency_type))
|
||||
except Exception as e:
|
||||
statistics["total"] += 1
|
||||
statistics["fail"] += 1
|
||||
if str(type(e)) not in statistics:
|
||||
statistics[str(type(e))] = 0
|
||||
statistics[str(type(e))] += 1
|
||||
raise e
|
||||
statistics["total"] += 1
|
||||
statistics["success"] += 1
|
||||
statistics["avg_time_per_sample"] = str((datetime.now() - now) / statistics["success"])
|
||||
|
||||
def run(url, header, llm, temperature, top_p, check, sampler, tools, figure_dir, wf, statistics, now, dependency_type):
|
||||
method = random.choices(list(method_weights.keys()), weights=list(method_weights.values()))[0]
|
||||
if method == "single":
|
||||
tool_number = 1
|
||||
else:
|
||||
tool_number = random.choices(list(number_weights.keys()), weights=list(number_weights.values()))[0]
|
||||
if method == "dag":
|
||||
tool_number = max(tool_number, 3)
|
||||
sample_with_statistics(url, header, llm, temperature, top_p, check, tool_number, sampler, tools, method, figure_dir, wf, statistics, now, dependency_type)
|
||||
|
||||
tasks = []
|
||||
for _ in range(number_of_samples):
|
||||
tasks.append(executor.submit(run, url, header, llm, temperature, top_p, check, sampler, tools, figure_dir, wf, statistics, now, dependency_type))
|
||||
for future in as_completed(tasks):
|
||||
try:
|
||||
future.result()
|
||||
done.append(future)
|
||||
except Exception as e:
|
||||
failed.append(future)
|
||||
|
||||
statistics_wf.write(json.dumps(statistics) + "\n")
|
||||
logger.info(f"Done: {len(done)}, Failed: {len(failed)}")
|
||||
|
||||
class RateLimitError(Exception):
|
||||
def __init__(self, message):
|
||||
super().__init__(message)
|
||||
|
||||
class ContentFormatError(Exception):
|
||||
def __init__(self, message):
|
||||
super().__init__(message)
|
||||
|
||||
async def sample(url, header, llm, temperature, top_p, check, tool_number, sampler, tools, method, figure_dir, wf, dependency_type):
|
||||
start_time = datetime.now()
|
||||
sample_id = str(uuid.uuid4().int)[:8]
|
||||
sub_G = sampler.sample_subgraph(tool_number, sample_method=method)
|
||||
|
||||
tool_list = list(sub_G.nodes)
|
||||
tool_edge = list(sub_G.edges)
|
||||
seed = random.randint(0, 1000000)
|
||||
sampled_tools_string = "Given a tool graph with tools as nodes, and invoking chains between tools as edges. The following tools (nodes) are available with their corresponding descriptions and input/outputs types:\n"
|
||||
for k, tool in enumerate(tool_list):
|
||||
sampled_tools_string += f"Node {k+1}:" + json.dumps(tools[tool]) + "\n"
|
||||
|
||||
sampled_links_string = "These tools can be connected as follows (the directed edges are invoking chains among tools):\n"
|
||||
for k, edge in enumerate(tool_edge):
|
||||
sampled_links_string += f"Edge: " + edge[0] + " -> " + edge[1] + "\n"
|
||||
prompt = """\nBased on the above tool graph, please be skillful to generate the according task steps, user request and tool invoking graph. \nRequirements: \n1. the generated user request should be somewhat clear, self-contained (user-specified text, image, video, audio, content should be contained in the request) and practical (help users solve a practical problem); \n2. the task steps must be strictly aligned with the tool graph (nodes and edges) and reasonable, the tool invoking graph must align with task steps, also with the given tool graph; \n3. the user request just can be decomposed into task steps solved by the tool invoking graph; \n4. each task step corresponds to a tool node in the tool graph and tool invoking graph, and the number of task steps must be same with the nodes. Each tool node can only be used once; \n5. if need image/audio/video resources in user request, please use files 'example.[jpg/mp4/wav/png]'; \n6. the dependencies among task steps must align with the edges of tool graph and tool invoking graph; \n7. the number and types of tool parameters in the generated tool invoking graph need to be consistent with the pre-defined input/outputs types of the tools. \nNow please generate your result (with random seed {""" + f"{seed}"+ """}) in a compact JSON format"""
|
||||
if dependency_type == "resource":
|
||||
prompt += """{"task_steps": [ step description of one or more steps ], "user_request": "your high-quality and self-contained synthesized request", "invoking_graph": {"nodes": [{"id": "tool name", "input": [ either user-specified text or resource file 'example.[jpg/mp4/wav/png' ] in the above user request, or the dependent tool name whose output is required by this node ]}], "links": [{"source": "tool name i", "target": "tool name j"}]}"""
|
||||
else:
|
||||
prompt += """{"task_steps": [ "concrete steps, format as Step x: Call xxx tool with xxx: 'xxx' and xxx: 'xxx'" ], "user_request": "your high-quality, concrete and self-contained synthesized request, with explicit parameter values", "invoking_graph": {"nodes": [{"id": "tool name", "arguments": [ {"name": "parameter name", "value": "parameter value, either user-specified text or the specific name of the tool whose result is required by this node"} ]}], "links": [{"source": "tool name i", "target": "tool name j"}]}"""
|
||||
if check:
|
||||
prompt += """, "check_by_teacher": "This field is filled by your strict and well-trained teacher, minor mistakes are complete intolerable to him. He evaluated whether your synthesized user request, tool invoking graph are valid and whether they are aligned with the given tool graph (strictly checked step by step according to the above requirements). Some comments from him place here (start with 'Let me check your result step by step, and evaluate the 'Executable' and 'Correct' of the tool invoking graph (Executable means that the tool invoking graph executed successfully, regardless of alignment with the given tool graph. While Correct implies that the tool invoking graph are not only 'Executable' but also strictly consistent (with strictly same nodes and same edges) with the given tool graph). After carefully evaluating, found some mistakes:' and end with a conclusion: 'Conclusion: Executable: no/yes, Correct: no/yes'.)"""
|
||||
prompt += "}:"
|
||||
|
||||
final_prompt = sampled_tools_string + sampled_links_string + prompt
|
||||
|
||||
if dependency_type == "temporal":
|
||||
final_prompt = final_prompt.replace("tool", "API")
|
||||
|
||||
payload = json.dumps({
|
||||
"model": f"{llm}",
|
||||
"messages": [
|
||||
{
|
||||
"role": "user",
|
||||
"content": final_prompt
|
||||
}
|
||||
],
|
||||
"temperature": temperature,
|
||||
"top_p": top_p,
|
||||
"frequency_penalty": 0,
|
||||
"presence_penalty": 0,
|
||||
"max_tokens": 2500,
|
||||
"stream": False,
|
||||
"stop": None
|
||||
})
|
||||
try:
|
||||
async with aiohttp.ClientSession() as session:
|
||||
async with session.post(url, headers=header, data=payload, timeout=120) as response:
|
||||
resp = await response.json()
|
||||
|
||||
if response.status == 429:
|
||||
raise RateLimitError(f"{resp}")
|
||||
if response.status != 200:
|
||||
raise Exception(f"{resp}")
|
||||
|
||||
content = resp["choices"][0]["message"]["content"]
|
||||
content = content.replace("\n", "")
|
||||
json_start = 0
|
||||
json_end = len(content)
|
||||
|
||||
if "```json" in content:
|
||||
json_start = content.find("```json") + 7
|
||||
if "```" in content:
|
||||
json_end = content.rfind("```")
|
||||
content = content[json_start:json_end]
|
||||
logger.info(content)
|
||||
try:
|
||||
content = json.loads(content)
|
||||
except json.JSONDecodeError as e:
|
||||
raise ContentFormatError(f"{content}")
|
||||
|
||||
if dependency_type == "resource":
|
||||
sampled_nodes = [{"id": tool, "input-type": tools[tool]["input-type"], "output-type": tools[tool]["output-type"]} for tool in tool_list]
|
||||
else:
|
||||
sampled_nodes = [{"id": tool, "parameters": tools[tool]["parameters"]} for tool in tool_list]
|
||||
|
||||
sampled_links = [{"source": edge[0], "target": edge[1]} for edge in tool_edge]
|
||||
sampled_nodes = sorted(sampled_nodes, key=lambda x: x["id"])
|
||||
sampled_links = sorted(sampled_links, key=lambda x: (x["source"], x["target"]))
|
||||
|
||||
content["invoking_graph"]["nodes"] = sorted(content["invoking_graph"]["nodes"], key=lambda x: x["id"])
|
||||
content["invoking_graph"]["links"] = sorted(content["invoking_graph"]["links"], key=lambda x: (x["source"], x["target"]))
|
||||
|
||||
result = {"id": sample_id, "seed": seed, "method": method, "number_of_tools": tool_number, "sampled_nodes": sampled_nodes, "sampled_links": sampled_links, "result": content}
|
||||
|
||||
if wf:
|
||||
wf.write(json.dumps(result) + "\n")
|
||||
|
||||
if figure_dir:
|
||||
plt.figure()
|
||||
pos = nx.circular_layout(sub_G)
|
||||
nx.draw_networkx_nodes(sub_G, pos, node_color="skyblue", node_size=300)
|
||||
nx.draw_networkx_edges(sub_G, pos, arrows=True)
|
||||
nx.draw_networkx_labels(sub_G, pos, font_size=8)
|
||||
plt.tight_layout()
|
||||
plt.savefig(f"{figure_dir}/{sample_id}.jpg")
|
||||
plt.close()
|
||||
|
||||
sampled_nodes_ids = [node["id"] for node in sampled_nodes]
|
||||
generated_nodes_ids = [node["id"] for node in content["invoking_graph"]["nodes"]]
|
||||
|
||||
end_time = datetime.now()
|
||||
logger.info(f"Sample {sample_id} finished, time cost: {end_time - start_time}")
|
||||
if sampled_links == content["invoking_graph"]["links"] and sampled_nodes_ids == generated_nodes_ids:
|
||||
logger.info("Check success: invoking graph and sampled graph are aligned.")
|
||||
elif sampled_nodes_ids != generated_nodes_ids:
|
||||
logger.info("Check fail: mismatched nodes")
|
||||
logger.info("Sampled node:\n" + json.dumps(sampled_nodes_ids, indent=2))
|
||||
logger.info("Generated node:\n" + json.dumps(generated_nodes_ids, indent=2))
|
||||
logger.info(f"Sample {sample_id}:\n{json.dumps(result, indent=2)}")
|
||||
else:
|
||||
logger.info("Check fail: mismatched links")
|
||||
logger.info("Sampled link:\n" + json.dumps(sampled_links, indent=2))
|
||||
logger.info("Generated link:\n" + json.dumps(content["invoking_graph"]["links"], indent=2))
|
||||
logger.info(f"Sample {sample_id}:\n{json.dumps(result, indent=2)}")
|
||||
except Exception as e:
|
||||
logger.info(f"Failed: {type(e)}")
|
||||
print(traceback.format_exc())
|
||||
raise e
|
||||
return result
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,237 @@
|
||||
{
|
||||
"nodes": [
|
||||
{
|
||||
"id": "Token Classification",
|
||||
"desc": "Token classification is a natural language understanding task in which a label is assigned to some tokens in a text. Some popular token classification subtasks are Named Entity Recognition (NER) and Part-of-Speech (PoS) tagging. NER models could be trained to identify specific entities in a text, such as dates, individuals and places; and PoS tagging would identify, for example, which words in a text are verbs, nouns, and punctuation marks.",
|
||||
"input-type": [
|
||||
"text"
|
||||
],
|
||||
"output-type": [
|
||||
"text"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "Translation",
|
||||
"desc": "Translation is the task of converting text from one language to another.",
|
||||
"input-type": [
|
||||
"text"
|
||||
],
|
||||
"output-type": [
|
||||
"text"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "Summarization",
|
||||
"desc": "Summarization is the task of producing a shorter version of a document while preserving its important information. Some models can extract text from the original input, while other models can generate entirely new text.",
|
||||
"input-type": [
|
||||
"text"
|
||||
],
|
||||
"output-type": [
|
||||
"text"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "Question Answering",
|
||||
"desc": "Question Answering models can retrieve the answer to a question from a given text, which is useful for searching for an answer in a document.",
|
||||
"input-type": [
|
||||
"text",
|
||||
"text"
|
||||
],
|
||||
"output-type": [
|
||||
"text"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "Conversational",
|
||||
"desc": "Conversational response modelling is the task of generating conversational text that is relevant, coherent and knowledgable given a prompt. These models have applications in chatbots, and as a part of voice assistants",
|
||||
"input-type": [
|
||||
"text"
|
||||
],
|
||||
"output-type": [
|
||||
"text"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "Text Generation",
|
||||
"desc": "Generating text is the task of producing new text. These models can, for example, fill in incomplete text or paraphrase.",
|
||||
"input-type": [
|
||||
"text"
|
||||
],
|
||||
"output-type": [
|
||||
"text"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "Sentence Similarity",
|
||||
"desc": "Sentence Similarity is the task of determining how similar two texts are. This task is particularly useful for information retrieval and clustering/grouping.",
|
||||
"input-type": [
|
||||
"text",
|
||||
"text"
|
||||
],
|
||||
"output-type": []
|
||||
},
|
||||
{
|
||||
"id": "Tabular Classification",
|
||||
"desc": "Tabular classification is the task of classifying a table (in Image format).",
|
||||
"input-type": [
|
||||
"image"
|
||||
],
|
||||
"output-type": [
|
||||
"text"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "Object Detection",
|
||||
"desc": "Object Detection models allow users to identify objects of certain defined classes. Object detection models receive an image as input and output the images with bounding boxes and labels on detected objects.",
|
||||
"input-type": [
|
||||
"image"
|
||||
],
|
||||
"output-type": [
|
||||
"text"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "Image Classification",
|
||||
"desc": "Image classification is the task of assigning a label or class to an entire image. Images are expected to have only one class for each image. Image classification models take an image as input and return a prediction about which class the image belongs to.",
|
||||
"input-type": [
|
||||
"image"
|
||||
],
|
||||
"output-type": [
|
||||
"text"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "Image-to-Image",
|
||||
"desc": "Image-to-image is the task of transforming a source image to match the characteristics of a target image or a target image domain. Any image manipulation and enhancement is possible with image to image models.",
|
||||
"input-type": [
|
||||
"image"
|
||||
],
|
||||
"output-type": [
|
||||
"image"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "Image-to-Text",
|
||||
"desc": "Image to text models output a text from a given image. Image captioning or optical character recognition can be considered as the most common applications of image to text.",
|
||||
"input-type": [
|
||||
"image"
|
||||
],
|
||||
"output-type": [
|
||||
"text"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "Text-to-Image",
|
||||
"desc": "Generates images from input text. These models can be used to generate images based on text prompts.",
|
||||
"input-type": [
|
||||
"text"
|
||||
],
|
||||
"output-type": [
|
||||
"image"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "Text-to-Video",
|
||||
"desc": "Generates videos from input text. These models can be used to generate videos based on text prompts.",
|
||||
"input-type": [
|
||||
"text"
|
||||
],
|
||||
"output-type": [
|
||||
"video"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "Visual Question Answering",
|
||||
"desc": "Visual Question Answering is the task of answering questions based on an image.",
|
||||
"input-type": [
|
||||
"image",
|
||||
"text"
|
||||
],
|
||||
"output-type": [
|
||||
"text"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "Document Question Answering",
|
||||
"desc": "Document Question Answering (also known as Document Visual Question Answering) is the task of answering questions on document images. Document question answering models take a (document, question) pair as input and return an answer in natural language. Models usually rely on multi-modal features, combining text, position of words (bounding-boxes) and image.",
|
||||
"input-type": [
|
||||
"image",
|
||||
"text"
|
||||
],
|
||||
"output-type": [
|
||||
"text"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "Image Segmentation",
|
||||
"desc": "Image Segmentation divides an image into segments where each pixel in the image is mapped to an object. This task has multiple variants such as instance segmentation, panoptic segmentation and semantic segmentation.",
|
||||
"input-type": [
|
||||
"image"
|
||||
],
|
||||
"output-type": [
|
||||
"image"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "Depth Estimation",
|
||||
"desc": "Depth estimation is the task of predicting depth of the objects present in an image.",
|
||||
"input-type": [
|
||||
"image"
|
||||
],
|
||||
"output-type": [
|
||||
"image"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "Text-to-Speech",
|
||||
"desc": "Text-to-Speech (TTS) is the task of generating natural sounding speech given text input. TTS models can be extended to have a single model that generates speech for multiple speakers and multiple languages.",
|
||||
"input-type": [
|
||||
"text"
|
||||
],
|
||||
"output-type": [
|
||||
"audio"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "Automatic Speech Recognition",
|
||||
"desc": "Automatic Speech Recognition (ASR), also known as Speech to Text (STT), is the task of transcribing a given audio to text. It has many applications, such as voice user interfaces.",
|
||||
"input-type": [
|
||||
"audio"
|
||||
],
|
||||
"output-type": [
|
||||
"text"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "Audio-to-Audio",
|
||||
"desc": "Audio-to-Audio is a family of tasks in which the input is an audio and the output is one or multiple generated audios. Some example tasks are speech enhancement and source separation.",
|
||||
"input-type": [
|
||||
"audio"
|
||||
],
|
||||
"output-type": [
|
||||
"audio"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "Audio Classification",
|
||||
"desc": "Audio classification is the task of assigning a label or class to a given audio. It can be used for recognizing which command a user is giving or the emotion of a statement, as well as identifying a speaker.",
|
||||
"input-type": [
|
||||
"audio"
|
||||
],
|
||||
"output-type": [
|
||||
"text"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "Image Editing",
|
||||
"desc": "Image editing is the task of modifying an image to match a given text description. It can be used to modify the attributes of an image, such as the color of an object or the background.",
|
||||
"input-type": [
|
||||
"text",
|
||||
"image"
|
||||
],
|
||||
"output-type": [
|
||||
"image"
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,411 @@
|
||||
{
|
||||
"nodes": [
|
||||
{
|
||||
"id": "Image Downloader",
|
||||
"desc": "Downloads an image from a given URL.",
|
||||
"input-type": [
|
||||
"url"
|
||||
],
|
||||
"output-type": [
|
||||
"image"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "Video Downloader",
|
||||
"desc": "Downloads a video from a given URL.",
|
||||
"input-type": [
|
||||
"url"
|
||||
],
|
||||
"output-type": [
|
||||
"video"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "Audio Downloader",
|
||||
"desc": "Downloads an audio file from a given URL.",
|
||||
"input-type": [
|
||||
"url"
|
||||
],
|
||||
"output-type": [
|
||||
"audio"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "Text Downloader",
|
||||
"desc": "Downloads the text content from a given URL.",
|
||||
"input-type": [
|
||||
"url"
|
||||
],
|
||||
"output-type": [
|
||||
"text"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "Text Search",
|
||||
"desc": "Searches for a specific text or keyword on the internet.",
|
||||
"input-type": [
|
||||
"text"
|
||||
],
|
||||
"output-type": [
|
||||
"text"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "Image Search",
|
||||
"desc": "Searches for images on the internet based on a given query.",
|
||||
"input-type": [
|
||||
"text"
|
||||
],
|
||||
"output-type": [
|
||||
"Image"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "Image Search (by Image)",
|
||||
"desc": "Performs a similar image search using an input image.",
|
||||
"input-type": [
|
||||
"image"
|
||||
],
|
||||
"output-type": [
|
||||
"image"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "URL Extractor",
|
||||
"desc": "Extracts URL from text",
|
||||
"input-type": [
|
||||
"text"
|
||||
],
|
||||
"output-type": [
|
||||
"url"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "Video Search",
|
||||
"desc": "Searches for videos on the internet based on a given query.",
|
||||
"input-type": [
|
||||
"text"
|
||||
],
|
||||
"output-type": [
|
||||
"video"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "Text-to-Image",
|
||||
"desc": "Generates an image based on a given text description.",
|
||||
"input-type": [
|
||||
"text"
|
||||
],
|
||||
"output-type": [
|
||||
"image"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "Text-to-Video",
|
||||
"desc": "Generates a video based on a given text description.",
|
||||
"input-type": [
|
||||
"text"
|
||||
],
|
||||
"output-type": [
|
||||
"video"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "Text-to-Audio",
|
||||
"desc": "Generates an audio file based on a given text description.",
|
||||
"input-type": [
|
||||
"text"
|
||||
],
|
||||
"output-type": [
|
||||
"audio"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "Image-to-Text",
|
||||
"desc": "Extracts text from an input image using Optical Character Recognition (OCR).",
|
||||
"input-type": [
|
||||
"image"
|
||||
],
|
||||
"output-type": [
|
||||
"text"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "Audio-to-Text",
|
||||
"desc": "Transcribes speech from an audio file into text.",
|
||||
"input-type": [
|
||||
"audio"
|
||||
],
|
||||
"output-type": [
|
||||
"text"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "Video-to-Text",
|
||||
"desc": "Transcribes speech from a video file into text.",
|
||||
"input-type": [
|
||||
"video"
|
||||
],
|
||||
"output-type": [
|
||||
"text"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "Audio Noise Reduction",
|
||||
"desc": "Reduces background noise or unwanted sounds from a given audio file.",
|
||||
"input-type": [
|
||||
"audio"
|
||||
],
|
||||
"output-type": [
|
||||
"audio"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "Audio Effects",
|
||||
"desc": "Applies various audio effects to a given audio file according to human instruction, such as reverb, chorus, or equalization.",
|
||||
"input-type": [
|
||||
"audio",
|
||||
"text"
|
||||
],
|
||||
"output-type": [
|
||||
"audio"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "Audio Splicer",
|
||||
"desc": "Combines two audio files into a single output file.",
|
||||
"input-type": [
|
||||
"audio",
|
||||
"audio"
|
||||
],
|
||||
"output-type": [
|
||||
"audio"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "Voice Changer",
|
||||
"desc": "Modifies the characteristics of a recorded voice according to human instruction, such as tone, pitch, or gender.",
|
||||
"input-type": [
|
||||
"audio",
|
||||
"text"
|
||||
],
|
||||
"output-type": [
|
||||
"audio"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "Text Summarizer",
|
||||
"desc": "Summarizes a given text into a shorter version while retaining the main points.",
|
||||
"input-type": [
|
||||
"text"
|
||||
],
|
||||
"output-type": [
|
||||
"text"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "Text Translator",
|
||||
"desc": "Translates a given text from one language to english.",
|
||||
"input-type": [
|
||||
"text"
|
||||
],
|
||||
"output-type": [
|
||||
"text"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "Text Sentiment Analysis",
|
||||
"desc": "Analyzes the sentiment of a given text, identifying if it is positive, negative, or neutral.",
|
||||
"input-type": [
|
||||
"text"
|
||||
],
|
||||
"output-type": [
|
||||
"text"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "Text Grammar Checker",
|
||||
"desc": "Checks a given text for grammatical errors and suggests corrections.",
|
||||
"input-type": [
|
||||
"text"
|
||||
],
|
||||
"output-type": [
|
||||
"text"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "Text Simplifier",
|
||||
"desc": "Rewrites a given text in a simpler and more understandable manner.",
|
||||
"input-type": [
|
||||
"text"
|
||||
],
|
||||
"output-type": [
|
||||
"text"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "Text Expander",
|
||||
"desc": "Expands a given short text into a more detailed and descriptive version.",
|
||||
"input-type": [
|
||||
"text"
|
||||
],
|
||||
"output-type": [
|
||||
"text"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "Keyword Extractor",
|
||||
"desc": "Extracts the most important keywords and phrases from a given text.",
|
||||
"input-type": [
|
||||
"text"
|
||||
],
|
||||
"output-type": [
|
||||
"text"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "Text Paraphraser",
|
||||
"desc": "Rewrites a given text using different words while maintaining its original meaning.",
|
||||
"input-type": [
|
||||
"text"
|
||||
],
|
||||
"output-type": [
|
||||
"text"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "Article Spinner",
|
||||
"desc": "Rewrites a given article using synonyms and syntax changes to create a new, unique version.",
|
||||
"input-type": [
|
||||
"text"
|
||||
],
|
||||
"output-type": [
|
||||
"text"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "Topic Generator",
|
||||
"desc": "Generates a list of relevant topics or ideas based on a given input.",
|
||||
"input-type": [
|
||||
"text"
|
||||
],
|
||||
"output-type": [
|
||||
"text"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "Audio-to-Image",
|
||||
"desc": "Generates an image that visually represents a given audio, such as a waveform or spectrogram.",
|
||||
"input-type": [
|
||||
"audio"
|
||||
],
|
||||
"output-type": [
|
||||
"image"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "Image-to-Video",
|
||||
"desc": "Creates a slideshow video using two input images.",
|
||||
"input-type": [
|
||||
"image",
|
||||
"image"
|
||||
],
|
||||
"output-type": [
|
||||
"video"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "Video-to-Audio",
|
||||
"desc": "Extracts the audio track from a given video file.",
|
||||
"input-type": [
|
||||
"video"
|
||||
],
|
||||
"output-type": [
|
||||
"audio"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "Video-to-Image",
|
||||
"desc": "Extracts a still image from a given video.",
|
||||
"input-type": [
|
||||
"video"
|
||||
],
|
||||
"output-type": [
|
||||
"image"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "Image Stitcher",
|
||||
"desc": "Stitches together two input images to create a panorama or collage.",
|
||||
"input-type": [
|
||||
"image",
|
||||
"image"
|
||||
],
|
||||
"output-type": [
|
||||
"image"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "Image Colorizer",
|
||||
"desc": "Adds color to a black and white input image using deep learning techniques.",
|
||||
"input-type": [
|
||||
"image"
|
||||
],
|
||||
"output-type": [
|
||||
"image"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "Image Style Transfer",
|
||||
"desc": "Applies the visual style of one image (style) to the content of another image (content) using neural style transfer techniques.",
|
||||
"input-type": [
|
||||
"image", "image"
|
||||
],
|
||||
"output-type": [
|
||||
"image"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "Video Stabilizer",
|
||||
"desc": "Stabilizes a shaky input video to produce a smoother output video.",
|
||||
"input-type": [
|
||||
"video"
|
||||
],
|
||||
"output-type": [
|
||||
"video"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "Video Speed Changer",
|
||||
"desc": "Adjusts the playback speed of a given video according to human instruction, either speeding it up or slowing it down.",
|
||||
"input-type": [
|
||||
"video", "text"
|
||||
],
|
||||
"output-type": [
|
||||
"video"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "Video Synchronization",
|
||||
"desc": "Synchronizes the timing of an existing voiceover or audio file with the visuals of a given video.",
|
||||
"input-type": [
|
||||
"video",
|
||||
"audio"
|
||||
],
|
||||
"output-type": [
|
||||
"video"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "Video Voiceover",
|
||||
"desc": "Adds a voiceover to a given video using a provided script or text.",
|
||||
"input-type": [
|
||||
"video",
|
||||
"text"
|
||||
],
|
||||
"output-type": [
|
||||
"video"
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,553 @@
|
||||
import traceback
|
||||
import numpy as np
|
||||
from scipy.optimize import linear_sum_assignment
|
||||
import json
|
||||
import click
|
||||
from datasets import load_metric
|
||||
import Levenshtein
|
||||
from sklearn.metrics import precision_recall_fscore_support as prfs
|
||||
import warnings
|
||||
import logging
|
||||
import os
|
||||
import itertools
|
||||
|
||||
warnings.filterwarnings("ignore")
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
logger.setLevel(logging.INFO)
|
||||
logger.handlers = []
|
||||
|
||||
def sim(name_1, name_2):
|
||||
if name_1 == "<PAD>" or name_2 == "<PAD>":
|
||||
return 0
|
||||
return 1 if name_1 == name_2 else 0
|
||||
|
||||
def create_cost_matrix(graph_1, graph_2):
|
||||
nodes_1 = graph_1["nodes"]
|
||||
nodes_2 = graph_2["nodes"]
|
||||
|
||||
num_nodes_1 = len(nodes_1)
|
||||
num_nodes_2 = len(nodes_2)
|
||||
|
||||
nodes_similarity_matrix = np.zeros((num_nodes_1, num_nodes_2))
|
||||
|
||||
for i, node_1 in enumerate(graph_1["nodes"]):
|
||||
for j, node_2 in enumerate(graph_2["nodes"]):
|
||||
nodes_similarity_matrix[i, j] = sim(node_1, node_2) #
|
||||
|
||||
links_similarity_matrix = np.zeros((num_nodes_1, num_nodes_2))
|
||||
for link_1 in graph_1["links"]:
|
||||
for link_2 in graph_2["links"]:
|
||||
if link_1["source"] == link_2["source"] and link_1["target"] == link_2["target"]:
|
||||
try:
|
||||
i_index_1 = nodes_1.index(link_1["source"])
|
||||
i_index_2 = nodes_2.index(link_2["source"])
|
||||
j_index_1 = nodes_1.index(link_1["target"])
|
||||
j_index_2 = nodes_2.index(link_2["target"])
|
||||
except ValueError:
|
||||
continue
|
||||
links_similarity_matrix[i_index_1, i_index_2] += 1
|
||||
links_similarity_matrix[j_index_1, j_index_2] += 1
|
||||
|
||||
cost_matrix = 2 - nodes_similarity_matrix - 0.5 * links_similarity_matrix
|
||||
return cost_matrix
|
||||
|
||||
def compute_assignment_matrix(graph_1, graph_2):
|
||||
cost_matrix = create_cost_matrix(graph_1, graph_2)
|
||||
row_ind, col_ind = linear_sum_assignment(cost_matrix)
|
||||
return row_ind, col_ind, cost_matrix[row_ind, col_ind].sum()
|
||||
|
||||
def matching(graph_1, graph_2):
|
||||
indices_1, indices_2, total_cost = compute_assignment_matrix(graph_1, graph_2)
|
||||
return indices_1, indices_2
|
||||
|
||||
def ratio_levenshtein(x, y):
|
||||
assert len(x) == len(y)
|
||||
n = len(x)
|
||||
total = 0
|
||||
for i in range(n):
|
||||
total += Levenshtein.ratio(x[i], y[i])
|
||||
return total / n
|
||||
|
||||
|
||||
def flatten(gt, pred, types = None):
|
||||
assert len(gt) == len(pred)
|
||||
|
||||
gt_flat = []
|
||||
pred_flat = []
|
||||
|
||||
for (sample_gt, sample_pred) in zip(gt, pred):
|
||||
union = set()
|
||||
|
||||
union.update(sample_gt)
|
||||
union.update(sample_pred)
|
||||
|
||||
for s in union:
|
||||
if types:
|
||||
if s in types:
|
||||
if s in sample_gt:
|
||||
gt_flat.append(types.index(s)+1)
|
||||
else:
|
||||
gt_flat.append(0)
|
||||
|
||||
if s in sample_pred:
|
||||
pred_flat.append(types.index(s)+1)
|
||||
else:
|
||||
pred_flat.append(0)
|
||||
else:
|
||||
gt_flat.append(0)
|
||||
pred_flat.append(0)
|
||||
else:
|
||||
if s in sample_gt:
|
||||
gt_flat.append(1)
|
||||
else:
|
||||
gt_flat.append(0)
|
||||
|
||||
if s in sample_pred:
|
||||
pred_flat.append(1)
|
||||
else:
|
||||
pred_flat.append(0)
|
||||
return gt_flat, pred_flat
|
||||
|
||||
def print_results(per_type, micro, macro, types, result_dict = None):
|
||||
columns = ('type', 'precision', 'recall', 'f1-score', 'support')
|
||||
|
||||
row_fmt = "%30s" + (" %12s" * (len(columns) - 1))
|
||||
logger.info(row_fmt % columns)
|
||||
|
||||
metrics_per_type = []
|
||||
for i, t in enumerate(types):
|
||||
metrics = []
|
||||
for j in range(len(per_type)):
|
||||
metrics.append(per_type[j][i])
|
||||
metrics_per_type.append(metrics)
|
||||
|
||||
for m, t in zip(metrics_per_type, types):
|
||||
logger.info(row_fmt % get_row(m, t))
|
||||
if result_dict is not None:
|
||||
result_dict[t] = {}
|
||||
result_dict[t]["precision"] = m[0]
|
||||
result_dict[t]["recall"] = m[1]
|
||||
result_dict[t]["f1-score"] = m[2]
|
||||
result_dict[t]["support"] = int(m[3])
|
||||
|
||||
logger.info('')
|
||||
|
||||
# micro
|
||||
logger.info(row_fmt % get_row(micro, 'micro'))
|
||||
|
||||
# macro
|
||||
logger.info(row_fmt % get_row(macro, 'macro'))
|
||||
|
||||
def get_row(data, label):
|
||||
row = [label]
|
||||
for i in range(len(data) - 1):
|
||||
row.append("%.2f" % (data[i] * 100))
|
||||
row.append(data[3])
|
||||
return tuple(row)
|
||||
|
||||
def get_content_type(content):
|
||||
content = content.strip('\'')
|
||||
assert isinstance(content, str), content
|
||||
# image
|
||||
for ext in ["jpg", "png", "jpeg", "gif", "bmp", "tiff", "svg", "ico"]:
|
||||
if "."+ext in content:
|
||||
return "image"
|
||||
# audio
|
||||
for ext in ["mp3", "wav", "wma", "ogg", "aac", "flac", "aiff", "au"]:
|
||||
if "."+ext in content:
|
||||
return "audio"
|
||||
# video
|
||||
for ext in ["mp4", "avi", "mov", "flv", "wmv", "mkv", "webm", "m4v", "mpg", "mpeg"]:
|
||||
if "."+ext in content:
|
||||
return "video"
|
||||
return "text"
|
||||
|
||||
@click.command()
|
||||
@click.option("--data_dir", default="data_huggingface", help="The directory of the data.")
|
||||
@click.option("--prediction_dir", default="predictions", help="The directory of the data.")
|
||||
@click.option("--save_dir", default=None, help="The directory to save the evaluation results")
|
||||
@click.option("--alignment", default=None)
|
||||
@click.option("--splits", "-s", multiple=True, default=["overall"])
|
||||
@click.option("--n_tools", "-n", multiple=True, default=["overall"])
|
||||
@click.option("--mode", default="add")
|
||||
@click.option("--metric", "-m", multiple=True, default=["all"])
|
||||
@click.option("--llm", default="gpt-3.5-turbo")
|
||||
@click.option("--dependency_type", type=str, default="resource")
|
||||
@click.option("--prompting", default="cot")
|
||||
def main(data_dir, prediction_dir, save_dir, splits, n_tools, mode, metric, llm, dependency_type, alignment, prompting):
|
||||
assert dependency_type in ["resource", "temporal"], "Dependency type not supported"
|
||||
args = locals()
|
||||
|
||||
if save_dir is None:
|
||||
save_dir = prediction_dir.replace("predictions", "metrics")
|
||||
save_dir = save_dir + f"_alignment_{alignment}" if alignment is not None else save_dir
|
||||
|
||||
formatter = logging.Formatter(f'%(asctime)s - [ {llm} ] - %(levelname)s - %(message)s')
|
||||
if not os.path.exists(f'{data_dir}/{save_dir}'):
|
||||
os.makedirs(f'{data_dir}/{save_dir}')
|
||||
|
||||
metric_file = f'{data_dir}/{save_dir}/{llm}.json'
|
||||
if os.path.exists(metric_file):
|
||||
all_metric_dict = json.load(open(metric_file, "r"))
|
||||
else:
|
||||
all_metric_dict = {}
|
||||
|
||||
file_handler = logging.FileHandler(f'{data_dir}/{save_dir}/{llm}.log')
|
||||
stream_handler = logging.StreamHandler()
|
||||
|
||||
file_handler.setFormatter(formatter)
|
||||
stream_handler.setFormatter(formatter)
|
||||
|
||||
logger.addHandler(file_handler)
|
||||
logger.addHandler(stream_handler)
|
||||
|
||||
if "all" in metric:
|
||||
metric = ["f1", "ed", "link", "argument", "rouge", "bertscore"]
|
||||
if prompting != "cot":
|
||||
metric = ["f1", "ed", "link", "argument"]
|
||||
|
||||
logger.info(f"Starts with: {args}")
|
||||
|
||||
tool_desc = json.load(open(f"{data_dir}/tool_desc.json", "r"))
|
||||
tool_map = {tool["id"]: i+1 for i, tool in enumerate(tool_desc["nodes"])}
|
||||
tool_map_reverse = {i+1: tool["id"] for i, tool in enumerate(tool_desc["nodes"])}
|
||||
tool_map_reverse[0] = "NEGATIVE"
|
||||
tool_map["<PAD>"] = -1
|
||||
|
||||
tool_output_type_map = None
|
||||
if dependency_type == "resource":
|
||||
tool_output_type_map = {tool["id"]: tool["output-type"][0] if len(tool["output-type"]) else "none" for tool in tool_desc["nodes"]}
|
||||
|
||||
splits = list(splits)
|
||||
n_tools = list(n_tools)
|
||||
|
||||
if "all" in splits:
|
||||
splits = ["overall", "single", "chain", "dag", ]
|
||||
if "all" in n_tools:
|
||||
n_tools = ["overall"] + [str(i) for i in range(1, 11)]
|
||||
|
||||
group = []
|
||||
|
||||
if mode == "mul":
|
||||
for s in splits:
|
||||
for n in n_tools:
|
||||
if (s, n) not in group:
|
||||
group.append((s, n))
|
||||
elif mode == "add":
|
||||
for s in splits:
|
||||
if (s, "overall") not in group:
|
||||
group.append((s, "overall"))
|
||||
for n in n_tools:
|
||||
if ("overall", n) not in group:
|
||||
group.append(("overall", n))
|
||||
else:
|
||||
assert False, "mode should be mul or add"
|
||||
|
||||
for s, n in group:
|
||||
logger.info("-"*15)
|
||||
logger.info(f"Tools Number: {n}, Task Split: {s}")
|
||||
evaluate(data_dir, prediction_dir, llm, s, n, metric, tool_desc, tool_map, tool_output_type_map, tool_map_reverse, all_metric_dict, dependency_type=dependency_type, alignment=alignment)
|
||||
|
||||
metric_json = open(metric_file, "w")
|
||||
metric_json.write(json.dumps(all_metric_dict, indent=2))
|
||||
|
||||
def evaluate(data_dir, prediction_dir, llm, split, n_tool, metric, tool_desc, tool_map, tool_output_type_map, tool_map_reverse, all_metric_dict, dependency_type, alignment = None):
|
||||
if f"{split}_{n_tool}" in all_metric_dict:
|
||||
metric_dict = all_metric_dict[f"{split}_{n_tool}"]
|
||||
else:
|
||||
metric_dict = {}
|
||||
all_metric_dict[f"{split}_{n_tool}"] = metric_dict
|
||||
|
||||
label_rf = open(f"{data_dir}/data.json", "r")
|
||||
|
||||
alignment_ids = None
|
||||
if alignment is not None:
|
||||
if alignment == "human":
|
||||
label_rf = open(f"{data_dir}/data.json", "r")
|
||||
logger.info(f"Alignment Mode: {alignment} ({len(label_rf.readlines())})")
|
||||
else:
|
||||
alignment_file = open(f"{data_dir}/alignment_ids.json", "r")
|
||||
alignment_ids = json.load(alignment_file)
|
||||
alignment_ids = list(itertools.chain(*alignment_ids[f"{alignment}_alignment_id"].values()))
|
||||
logger.info(f"Alignment Mode: {alignment} ({len(alignment_ids)})")
|
||||
|
||||
predcition_rf = open(f"{data_dir}/{prediction_dir}/{llm}.json", "r")
|
||||
|
||||
predcitions = {}
|
||||
labels = {}
|
||||
label_rf = open(f"{data_dir}/data.json", "r")
|
||||
for line in label_rf:
|
||||
data = json.loads(line)
|
||||
real_tool_num = len(data["task_nodes"])
|
||||
if alignment_ids is None or data["id"] in alignment_ids:
|
||||
if split == "overall" or data["type"] == split:
|
||||
if n_tool == "overall" or str(real_tool_num) == n_tool:
|
||||
id = data["id"]
|
||||
labels[id] = data
|
||||
|
||||
for line in predcition_rf:
|
||||
try:
|
||||
data = json.loads(line)
|
||||
except Exception as e:
|
||||
print(e)
|
||||
print(line)
|
||||
exit()
|
||||
id = data["id"]
|
||||
predcitions[id] = data
|
||||
|
||||
ids = set(labels.keys()).intersection(set(predcitions.keys()))
|
||||
labels = {id: labels[id] for id in ids}
|
||||
predcitions = {id: predcitions[id] for id in ids}
|
||||
|
||||
predcition_task_steps = []
|
||||
label_task_steps = []
|
||||
predcition_names = []
|
||||
label_names = []
|
||||
label_graphs = []
|
||||
predcition_graphs = []
|
||||
label_links = []
|
||||
predcition_links = []
|
||||
label_task_arg_names = []
|
||||
predcition_task_arg_names = []
|
||||
label_task_arg_name_values = []
|
||||
predcition_task_arg_name_values = []
|
||||
|
||||
for id in ids:
|
||||
try:
|
||||
label = labels[id]
|
||||
predcition = predcitions[id]
|
||||
|
||||
if "rouge" in metric or "bertscore" in metric:
|
||||
predcition_task_step = predcition["result"]["task_steps"]
|
||||
label_task_step = label["task_steps"]
|
||||
|
||||
try:
|
||||
if isinstance(predcition_task_step[0], str):
|
||||
predcition_task_steps.append("\n".join(predcition_task_step))
|
||||
else:
|
||||
if "task" in predcition_task_step[0]:
|
||||
predcition_task_steps.append("\n".join([step["task"] for step in predcition_task_step]))
|
||||
elif "step" in predcition_task_step[0]:
|
||||
predcition_task_steps.append("\n".join([step["step"] for step in predcition_task_step]))
|
||||
elif "id" in predcition_task_step[0]:
|
||||
predcition_task_steps.append("\n".join([step["id"] for step in predcition_task_step]))
|
||||
elif "step_name" in predcition_task_step[0]:
|
||||
predcition_task_steps.append("\n".join([step["step_name"] for step in predcition_task_step]))
|
||||
else:
|
||||
predcition_task_steps.append("\n".join([step["description"] for step in predcition_task_step]))
|
||||
except Exception as e:
|
||||
predcition_task_steps.append(str(predcition_task_step))
|
||||
|
||||
label_task_steps.append("\n".join(label_task_step))
|
||||
|
||||
label_nodes = label["task_nodes"]
|
||||
predcition_nodes = predcition["result"]["task_nodes"]
|
||||
|
||||
label_node_name = [node["task"] for node in label_nodes]
|
||||
predcition_node_name = [node["task"] for node in predcition_nodes]
|
||||
|
||||
label_task_arg_name = []
|
||||
predcition_task_arg_name = []
|
||||
|
||||
label_task_arg_name_value = []
|
||||
predcition_task_arg_name_value = []
|
||||
|
||||
if dependency_type == "resource":
|
||||
predcition_node_name = [name.replace("_", " ") for name in predcition_node_name]
|
||||
label_node_name = [name.replace("_", " ") for name in label_node_name]
|
||||
label_link = []
|
||||
predcition_link = []
|
||||
for inx, node in enumerate(label_nodes):
|
||||
new_arguments = []
|
||||
for i, argument in enumerate(node["arguments"]):
|
||||
try:
|
||||
if isinstance(argument, dict):
|
||||
argument = list(argument.values())[0]
|
||||
if isinstance(argument, list):
|
||||
argument = " ".join(argument)
|
||||
if "<node-" in argument:
|
||||
index_start = argument.index("<node-") + 6
|
||||
index_end = argument.index(">")
|
||||
if int(argument[index_start: index_end]) == inx:
|
||||
continue
|
||||
argument_tool_name = label_node_name[int(argument[index_start: index_end])]
|
||||
label_link.append({"source": argument_tool_name, "target": node["task"]})
|
||||
new_argument = {"name": tool_output_type_map.get(argument_tool_name, "other"), "value": argument_tool_name}
|
||||
else:
|
||||
new_argument = {"name": get_content_type(argument), "value": argument}
|
||||
except Exception as e:
|
||||
pass
|
||||
new_arguments.append(new_argument)
|
||||
node["arguments"] = new_arguments
|
||||
|
||||
for inx, node in enumerate(predcition_nodes):
|
||||
new_arguments = []
|
||||
for i, argument in enumerate(node.get("arguments", [])):
|
||||
try:
|
||||
if isinstance(argument, dict):
|
||||
argument = list(argument.values())[0]
|
||||
if isinstance(argument, list):
|
||||
argument = " ".join(argument)
|
||||
if isinstance(argument, str) and "<node-" in argument:
|
||||
index_start = argument.index("<node-") + 6
|
||||
index_end = argument.index(">")
|
||||
|
||||
if int(argument[index_start: index_end]) == inx:
|
||||
continue
|
||||
prediction_tool_name = predcition_node_name[int(argument[index_start: index_end])]
|
||||
predcition_link.append({"source": prediction_tool_name, "target": node["task"]})
|
||||
new_argument = {"name": tool_output_type_map.get(prediction_tool_name, "other"), "value": prediction_tool_name}
|
||||
else:
|
||||
new_argument = {"name": get_content_type(argument), "value": argument}
|
||||
|
||||
except Exception as e:
|
||||
pass
|
||||
new_arguments.append(new_argument)
|
||||
node["arguments"] = new_arguments
|
||||
else:
|
||||
predcition_link = predcition["result"]["task_links"]
|
||||
label_link = label["task_links"]
|
||||
|
||||
predcition_node_argument = [node.get("arguments", []) for node in predcition_nodes]
|
||||
label_node_argument = [node["arguments"] for node in label_nodes]
|
||||
|
||||
for task, arguments in zip (predcition_node_name, predcition_node_argument):
|
||||
for argument in arguments:
|
||||
label_task_arg_name.append(f"{task}-{argument['name']}")
|
||||
label_task_arg_name_value.append(f"{task}-{argument['name']}-{argument['value']}")
|
||||
|
||||
for task, arguments in zip (label_node_name, label_node_argument):
|
||||
for argument in arguments:
|
||||
predcition_task_arg_name.append(f"{task}-{argument['name']}")
|
||||
predcition_task_arg_name_value.append(f"{task}-{argument['name']}-{argument['value']}")
|
||||
|
||||
label_graph = {
|
||||
"nodes": label_node_name,
|
||||
"links": label_link,
|
||||
"arguments": label_node_argument
|
||||
}
|
||||
predcition_graph = {
|
||||
"nodes": predcition_node_name,
|
||||
"links": predcition_link,
|
||||
"arguments": predcition_node_argument
|
||||
}
|
||||
|
||||
label_graphs.append(label_graph)
|
||||
predcition_graphs.append(predcition_graph)
|
||||
|
||||
for node_name in predcition_node_name:
|
||||
assert isinstance(node_name, str), node_name
|
||||
|
||||
predcition_names.append(predcition_node_name)
|
||||
label_names.append(label_node_name)
|
||||
|
||||
predcition_task_arg_names.append(predcition_task_arg_name)
|
||||
label_task_arg_names.append(label_task_arg_name)
|
||||
|
||||
predcition_task_arg_name_values.append(predcition_task_arg_name_value)
|
||||
label_task_arg_name_values.append(label_task_arg_name_value)
|
||||
|
||||
label_links.append(label_link)
|
||||
predcition_links.append(predcition_link)
|
||||
|
||||
except Exception as e:
|
||||
logger.info(f"Parsing Error: {e}, Ignore #id {id}")
|
||||
logger.info(traceback.format_exc())
|
||||
|
||||
logger.info(f"Step Supports: {len(label_task_steps)} / {len(ids)}")
|
||||
logger.info(f"Node Support: {len(label_names)} / {len(ids)}")
|
||||
logger.info(f"Link Support: {len(label_links)} / {len(ids)}")
|
||||
logger.info(f"Argument Support: {len(label_graphs)} / {len(ids)}")
|
||||
|
||||
metric_dict["all_samples"] = len(ids)
|
||||
metric_dict["step_supports"] = len(label_task_steps)
|
||||
metric_dict["node_supports"] = len(label_names)
|
||||
metric_dict["link_supports"] = len(label_links)
|
||||
metric_dict["argument_supports"] = len(label_graphs)
|
||||
|
||||
if len(label_graphs) == 0 or len(label_names) == 0 or len(label_links) == 0:
|
||||
logger.info("No supports, skip")
|
||||
return
|
||||
|
||||
if "rouge" in metric:
|
||||
rouge = load_metric("rouge")
|
||||
rouge_scores = rouge.compute(predictions=predcition_task_steps, references=label_task_steps, use_aggregator=True)
|
||||
for key in rouge_scores:
|
||||
logger.info(f"Step {key}: {rouge_scores[key].mid.fmeasure}")
|
||||
metric_dict[f"step_{key}"] = rouge_scores[key].mid.fmeasure
|
||||
|
||||
if "bertscore" in metric:
|
||||
bertscore = load_metric("bertscore")
|
||||
bertscore_scores = bertscore.compute(predictions=predcition_task_steps, references=label_task_steps, model_type="roberta-large")
|
||||
for key in bertscore_scores:
|
||||
if key in ["precision", "recall", "f1"]:
|
||||
bertscore_scores[key] = np.mean(bertscore_scores[key])
|
||||
logger.info(f"Step BERTScore {key}: {bertscore_scores[key]}")
|
||||
metric_dict[f"step_bertscore_{key}"] = bertscore_scores[key]
|
||||
|
||||
if "f1" in metric or "argument" in metric:
|
||||
types = list(range(1, len(tool_desc["nodes"])+1))
|
||||
types_name = [tool_map_reverse[i] for i in types]
|
||||
gt_flat, pred_flat = flatten(label_names, predcition_names, types = types_name)
|
||||
|
||||
per_type = prfs(gt_flat, pred_flat, labels=types, average=None)
|
||||
micro = prfs(gt_flat, pred_flat, labels=types, average='micro')[:-1]
|
||||
macro = prfs(gt_flat, pred_flat, labels=types, average='macro')[:-1]
|
||||
total_support = sum(per_type[-1])
|
||||
|
||||
logger.info(f"Node Micro Precision [ No Matching ]: {micro[0]}")
|
||||
logger.info(f"Node Micro Recall [ No Matching ]: {micro[1]}")
|
||||
logger.info(f"Node Micro F1 [ No Matching ]: {micro[2]}")
|
||||
logger.info(f"Node Macro Precision [ No Matching ]: {macro[0]}")
|
||||
logger.info(f"Node Macro Recall [ No Matching ]: {macro[1]}")
|
||||
logger.info(f"Node Macro F1 [ No Matching ]: {macro[2]}")
|
||||
logger.info("Node Detailed Report [ No Matching ]: ")
|
||||
metric_dict["node_micro_precision_no_matching"] = micro[0]
|
||||
metric_dict["node_micro_recall_no_matching"] = micro[1]
|
||||
metric_dict["node_micro_f1_no_matching"] = micro[2]
|
||||
metric_dict["node_macro_precision_no_matching"] = macro[0]
|
||||
metric_dict["node_macro_recall_no_matching"] = macro[1]
|
||||
metric_dict["node_macro_f1_no_matching"] = macro[2]
|
||||
|
||||
per_type_metric = {}
|
||||
metric_dict["node_per_type_no_matchcing"] = per_type_metric
|
||||
print_results(per_type, list(micro) + [total_support], list(macro) + [total_support], types_name, result_dict = per_type_metric)
|
||||
|
||||
|
||||
gt_flat, pred_flat = flatten(label_task_arg_names, predcition_task_arg_names)
|
||||
micro = prfs(gt_flat, pred_flat, average="binary")[:-1]
|
||||
logger.info(f"Argument Task-ArgName Binary F1: [ No Matching ]: {micro[-1]}")
|
||||
metric_dict["argument_task_argname_binary_f1_no_matching"] = micro[-1]
|
||||
|
||||
gt_flat, pred_flat = flatten(label_task_arg_name_values, predcition_task_arg_name_values)
|
||||
micro = prfs(gt_flat, pred_flat, average="binary")[:-1]
|
||||
logger.info(f"Argument Task-ArgName-Value Binary F1 [ No Matching ]: {micro[-1]}")
|
||||
metric_dict["argument_task_argname_value_binary_f1_no_matching"] = micro[-1]
|
||||
|
||||
if "ed" in metric:
|
||||
labels = []
|
||||
predcitions = []
|
||||
for label_name, predcition_name in zip(label_names, predcition_names):
|
||||
labels.append([tool_map.get(name, 0) for name in label_name])
|
||||
predcitions.append([tool_map.get(name, 0) for name in predcition_name])
|
||||
ed = ratio_levenshtein(predcitions, labels)
|
||||
logger.info(f"Edit Distance: {1-ed}")
|
||||
metric_dict["edit_distance"] = 1-ed
|
||||
|
||||
if "link" in metric:
|
||||
tuple_label_links = []
|
||||
tuple_predcition_links = []
|
||||
for label_link, predcition_link in zip(label_links, predcition_links):
|
||||
tuple_label_links.append([(link["source"], link["target"]) for link in label_link])
|
||||
tuple_predcition_links.append([(link["source"], link["target"]) for link in predcition_link])
|
||||
|
||||
gt_flat, pred_flat = flatten(tuple_label_links, tuple_predcition_links)
|
||||
|
||||
|
||||
micro = prfs(gt_flat, pred_flat, average="binary")[:-1]
|
||||
logger.info(f"Link Binary F1: {micro[-1]}")
|
||||
metric_dict["link_binary_f1"] = micro[-1]
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,121 @@
|
||||
import json
|
||||
import click
|
||||
import traceback
|
||||
|
||||
def formulate_sample(data, dependency_type):
|
||||
try:
|
||||
user_request = data["result"]["user_request"]
|
||||
invoking_graph = data["result"]["invoking_graph"]
|
||||
task_steps = data["result"]["task_steps"]
|
||||
nodes = invoking_graph["nodes"]
|
||||
links = invoking_graph["links"]
|
||||
user_request = data["result"]["user_request"]
|
||||
if "check_by_teacher" in data["result"]:
|
||||
check_by_teacher = data["result"]["check_by_teacher"]
|
||||
else:
|
||||
check_by_teacher = invoking_graph["check_by_teacher"]
|
||||
for node in nodes:
|
||||
node["task"] = node["id"]
|
||||
node.pop("id")
|
||||
if dependency_type == "resource":
|
||||
node["task"] = node["task"].replace("_", " ")
|
||||
node["arguments"] = node["input"]
|
||||
node.pop("input")
|
||||
|
||||
for node in nodes:
|
||||
assert isinstance(node, dict)
|
||||
assert "task" in node
|
||||
assert "arguments" in node
|
||||
if isinstance(node["arguments"], str) and node["arguments"].startswith("<node-"):
|
||||
node["arguments"] = [node["arguments"]]
|
||||
assert isinstance(node["arguments"], list), node["arguments"]
|
||||
if dependency_type == "resource":
|
||||
assert len(node["arguments"]) <= 2
|
||||
for i, argument in enumerate(node["arguments"]):
|
||||
for j, n in enumerate(nodes):
|
||||
if n["task"] in argument:
|
||||
node["arguments"][i] = f"<node-{j}>"
|
||||
break
|
||||
for link in links:
|
||||
assert isinstance(link, dict)
|
||||
assert "source" in link
|
||||
assert "target" in link
|
||||
if dependency_type == "resource":
|
||||
link["source"] = link["source"].replace("_", " ")
|
||||
link["target"] = link["target"].replace("_", " ")
|
||||
assert isinstance(task_steps, list)
|
||||
assert isinstance(nodes, list)
|
||||
assert len(nodes) == len(task_steps)
|
||||
assert isinstance(user_request, str)
|
||||
assert isinstance(check_by_teacher, str)
|
||||
except Exception as e:
|
||||
print(e)
|
||||
traceback.print_exc()
|
||||
return None, None, None, None, None
|
||||
return user_request, task_steps, links, nodes, check_by_teacher
|
||||
|
||||
@click.command()
|
||||
@click.option('--data_dir', default='data_huggingface', help='Path to the data directory')
|
||||
@click.option('--dependency_type', default='resource')
|
||||
def formulate(data_dir, dependency_type):
|
||||
rf = open(f"{data_dir}/data_raw.json", "r")
|
||||
wf_format = open(f"{data_dir}/data.json", "w")
|
||||
wf_error = open(f"{data_dir}/data_error.json", "w")
|
||||
wf_ur = open(f"{data_dir}/user_requests.json", "w")
|
||||
|
||||
all = 0
|
||||
format = 0
|
||||
for line in rf:
|
||||
all += 1
|
||||
data = json.loads(line)
|
||||
method = data["method"]
|
||||
n_tools = data["number_of_tools"]
|
||||
seed = data["seed"]
|
||||
_id = data["id"]
|
||||
sampled_nodes = data["sampled_nodes"]
|
||||
sampled_links = data["sampled_links"]
|
||||
for sampled_node in sampled_nodes:
|
||||
sampled_node["task"] = sampled_node["id"]
|
||||
sampled_node.pop("id")
|
||||
if dependency_type == "resource":
|
||||
sampled_node["task"] = sampled_node["task"].replace("_", " ")
|
||||
if "input" in sampled_node:
|
||||
sampled_node["input-type"] = sampled_node["input"]
|
||||
sampled_node.pop("input")
|
||||
sampled_node["output-type"] = sampled_node["output"]
|
||||
sampled_node.pop("output")
|
||||
else:
|
||||
sampled_node["arguments"] = sampled_node["parameters"]
|
||||
sampled_node.pop("parameters")
|
||||
user_request, task_steps, links, nodes, check_by_teacher = formulate_sample(data, dependency_type)
|
||||
if user_request is None:
|
||||
wf_error.write(line)
|
||||
continue
|
||||
format += 1
|
||||
result = {
|
||||
"id": _id,
|
||||
"seed": seed,
|
||||
"type": method,
|
||||
"n_tools": n_tools,
|
||||
"sampled_nodes": sampled_nodes,
|
||||
"sampled_links": sampled_links,
|
||||
"user_request": user_request,
|
||||
"task_steps": task_steps,
|
||||
"task_nodes": nodes,
|
||||
"task_links": links,
|
||||
"check_by_teacher": check_by_teacher,
|
||||
}
|
||||
wf_format.write(json.dumps(result)+"\n")
|
||||
ur_result = {
|
||||
"id": _id,
|
||||
"user_request": user_request,
|
||||
}
|
||||
wf_ur.write(json.dumps(ur_result)+"\n")
|
||||
wf_format.close()
|
||||
wf_error.close()
|
||||
wf_ur.close()
|
||||
rf.close()
|
||||
print(f"Format {format} out of {all}")
|
||||
|
||||
if __name__ == "__main__":
|
||||
formulate()
|
||||
@@ -0,0 +1,62 @@
|
||||
import json
|
||||
import click
|
||||
|
||||
def generate_graph_resource(tool_file):
|
||||
with open(tool_file) as f:
|
||||
data = json.load(f)
|
||||
data = data["nodes"]
|
||||
assert "input-type" in data[0] and "output-type" in data[0], "Input and output types are not defined"
|
||||
nodes = []
|
||||
for i in range(len(data)):
|
||||
nodes.append({"id": data[i]["id"], "desc": data[i]["desc"], "input-type": data[i]["input-type"], "output-type": data[i]["output-type"]})
|
||||
links = []
|
||||
for i in range(len(nodes)):
|
||||
for j in range(len(nodes)):
|
||||
if i != j:
|
||||
if len(set(nodes[i]["output-type"]).intersection(set(nodes[j]["input-type"]))) > 0:
|
||||
links.append({"source": nodes[i]["id"], "target": nodes[j]["id"], "type": list(set(nodes[i]["output-type"]).intersection(set(nodes[j]["input-type"])))[0]})
|
||||
graph = {"nodes": nodes, "links": links}
|
||||
with open(tool_file.replace("tools", "graph"), 'w') as f:
|
||||
json.dump(graph, f, indent=2)
|
||||
|
||||
def generate_graph_temporal(tool_file):
|
||||
with open(tool_file) as f:
|
||||
data = json.load(f)
|
||||
nodes = []
|
||||
data = data["nodes"]
|
||||
if "parameters" not in data[0] and "input-type" not in data[0]:
|
||||
for i in range(len(data)):
|
||||
nodes.append({"id": data[i]["id"], "desc": data[i]["desc"]})
|
||||
elif "input-type" not in data[0]:
|
||||
for i in range(len(data)):
|
||||
nodes.append({"id": data[i]["id"], "desc": data[i]["desc"], "parameters": data[i]["parameters"]})
|
||||
else:
|
||||
for i in range(len(data)):
|
||||
nodes.append({"id": data[i]["id"], "desc": data[i]["desc"], "parameters": data[i]["parameters"], "input-type": data[i]["input-type"], "output-type": data[i]["output-type"]})
|
||||
links = []
|
||||
for i in range(len(nodes)):
|
||||
for j in range(len(nodes)):
|
||||
if i != j:
|
||||
links.append({"source": nodes[i]["id"], "target": nodes[j]["id"], "type": "complete"})
|
||||
graph = {"nodes": nodes, "links": links}
|
||||
with open(tool_file.replace("tools", "graph"), 'w') as f:
|
||||
json.dump(graph, f, indent=2)
|
||||
|
||||
@click.command()
|
||||
@click.option('--data_dir')
|
||||
@click.option('--tool_desc', default=None, help='Path to the tool description file')
|
||||
@click.option('--dependency_type', default='resource', help='Type of graph to generate')
|
||||
def generate_graph(tool_desc, data_dir, dependency_type):
|
||||
if tool_desc:
|
||||
tool_file = tool_desc
|
||||
else:
|
||||
tool_file = f"{data_dir}/graph_desc.json"
|
||||
if dependency_type == "temporal":
|
||||
generate_graph_temporal(tool_file)
|
||||
elif dependency_type == "resource":
|
||||
generate_graph_resource(tool_file)
|
||||
else:
|
||||
print("Type not supported")
|
||||
|
||||
if __name__ == "__main__":
|
||||
generate_graph()
|
||||
@@ -0,0 +1,233 @@
|
||||
import networkx as nx
|
||||
import random
|
||||
import json
|
||||
import matplotlib.pyplot as plt
|
||||
import click
|
||||
|
||||
random.seed(0)
|
||||
class GraphSampler:
|
||||
def __init__(self, graph: nx.Graph = None, file_name = None):
|
||||
if file_name:
|
||||
with open(file_name, "r") as f:
|
||||
data = json.load(f)
|
||||
|
||||
# Represent your graph in NetworkX
|
||||
graph = nx.DiGraph()
|
||||
|
||||
# Add nodes to the graph
|
||||
if "input-type" in data["nodes"][0]:
|
||||
for node in data["nodes"]:
|
||||
graph.add_node(node["id"], desc=node["desc"], input_type=node["input-type"], output_type=node["output-type"])
|
||||
else:
|
||||
for node in data["nodes"]:
|
||||
graph.add_node(node["id"], desc=node["desc"], parameters=node["parameters"])
|
||||
|
||||
# Add edges to the graph
|
||||
for link in data["links"]:
|
||||
graph.add_edge(link["source"], link["target"], type=link["type"])
|
||||
|
||||
self.graph = graph
|
||||
|
||||
def sample_subgraph_by_weight(self, number_weights, method_weights):
|
||||
method = random.choices(list(method_weights.keys()), weights=list(method_weights.values()))[0]
|
||||
if method == "single":
|
||||
tool_number = 1
|
||||
else:
|
||||
tool_number = random.choices(list(number_weights.keys()), weights=list(number_weights.values()))[0]
|
||||
return self.sample_subgraph(tool_number, sample_method=method)
|
||||
|
||||
def sample_subgraph(self, num_nodes=3, sample_method="chain"):
|
||||
seed_node = random.choice(list(self.graph.nodes))
|
||||
if sample_method == "single":
|
||||
sub_G = nx.DiGraph()
|
||||
sub_G.add_node(seed_node)
|
||||
return sub_G
|
||||
elif sample_method == "chain":
|
||||
return self.sample_subgraph_chain(seed_node, num_nodes)
|
||||
elif sample_method == "dag":
|
||||
return self.sample_subgraph_dag(seed_node, num_nodes)
|
||||
else:
|
||||
raise ValueError("Invalid sample method")
|
||||
|
||||
def sample_subgraph_chain(self, seed_node, num_nodes):
|
||||
# Create a list to store the sub-graph nodes
|
||||
sub_graph_nodes = [seed_node]
|
||||
head_node = seed_node
|
||||
tail_node = seed_node
|
||||
edges = []
|
||||
|
||||
# Keep adding nodes until we reach the desired number
|
||||
while len(sub_graph_nodes) < num_nodes:
|
||||
# Get the neighbors of the last node in the sub-graph
|
||||
head_node_neighbors = list(self.graph.predecessors(head_node))
|
||||
tail_node_neighbors = list(self.graph.successors(tail_node))
|
||||
neighbors = head_node_neighbors + tail_node_neighbors
|
||||
|
||||
# If the node has neighbors, randomly select one and add it to the sub-graph
|
||||
if len(neighbors) > 0:
|
||||
neighbor = random.choice(neighbors)
|
||||
if neighbor not in sub_graph_nodes:
|
||||
if neighbor in head_node_neighbors:
|
||||
sub_graph_nodes.insert(0, neighbor)
|
||||
edges.insert(0, (neighbor, head_node))
|
||||
head_node = neighbor
|
||||
else:
|
||||
sub_graph_nodes.append(neighbor)
|
||||
edges.append((tail_node, neighbor))
|
||||
tail_node = neighbor
|
||||
else:
|
||||
break
|
||||
|
||||
# Create the sub-graph
|
||||
sub_G = nx.DiGraph()
|
||||
sub_G.add_nodes_from(sub_graph_nodes)
|
||||
sub_G.add_edges_from(edges)
|
||||
|
||||
return sub_G
|
||||
|
||||
def sample_subgraph_dag(self, seed_node, num_nodes):
|
||||
# Create a list to store the sub-graph nodes
|
||||
sub_graph_nodes = [seed_node]
|
||||
edges = []
|
||||
|
||||
# Keep adding nodes until we reach the desired number
|
||||
while len(sub_graph_nodes) < num_nodes:
|
||||
# Randomly select a node from the current sub-graph
|
||||
node = random.choice(sub_graph_nodes)
|
||||
# prec_neighbors = list(self.graph.predecessors(node))
|
||||
succ_neighbors = list(self.graph.successors(node))
|
||||
|
||||
if "input_type" in self.graph.nodes[node]:
|
||||
# filter exisiting income edge type
|
||||
prec_neighbors = []
|
||||
input_type = list(self.graph.nodes[node]["input_type"])
|
||||
all_in_edges = list(self.graph.in_edges(node, data=True))
|
||||
for edge in edges:
|
||||
for ref_edge in all_in_edges:
|
||||
if edge[0] == ref_edge[0] and edge[1] == ref_edge[1]:
|
||||
input_type.remove(ref_edge[2]["type"])
|
||||
for edge in all_in_edges:
|
||||
if edge[2]["type"] in input_type:
|
||||
prec_neighbors.append(edge[0])
|
||||
else:
|
||||
prec_neighbors = list(self.graph.predecessors(node))
|
||||
|
||||
neighbors = prec_neighbors + succ_neighbors
|
||||
|
||||
# If the node has neighbors, randomly select one and add it to the sub-graph
|
||||
if neighbors:
|
||||
neighbor = random.choice(neighbors)
|
||||
if neighbor not in sub_graph_nodes:
|
||||
if neighbor in prec_neighbors:
|
||||
edges.append((neighbor, node))
|
||||
else:
|
||||
edges.append((node, neighbor))
|
||||
sub_graph_nodes.append(neighbor)
|
||||
# If the node has no neighbors, select a new node from the original graph
|
||||
else:
|
||||
node = random.choice(list(self.graph.nodes))
|
||||
if node not in sub_graph_nodes:
|
||||
sub_graph_nodes.append(node)
|
||||
|
||||
# Create the sub-graph
|
||||
sub_G = nx.DiGraph()
|
||||
sub_G.add_nodes_from(sub_graph_nodes)
|
||||
sub_G.add_edges_from(edges)
|
||||
|
||||
return sub_G
|
||||
|
||||
def sample_subgraph_random_walk(self, seed_node, num_nodes):
|
||||
# Create a list to store the sub-graph nodes
|
||||
sub_graph_nodes = [seed_node]
|
||||
edges = []
|
||||
|
||||
# Keep adding nodes until we reach the desired number
|
||||
while len(sub_graph_nodes) < num_nodes:
|
||||
# Randomly select a node from the current sub-graph
|
||||
node = random.choice(sub_graph_nodes)
|
||||
neighbors = list(self.graph.successors(node))
|
||||
|
||||
# If the node has neighbors, randomly select one and add it to the sub-graph
|
||||
if neighbors:
|
||||
neighbor = random.choice(neighbors)
|
||||
if neighbor not in sub_graph_nodes:
|
||||
edges.append((node, neighbor))
|
||||
sub_graph_nodes.append(neighbor)
|
||||
# If the node has no neighbors, select a new node from the original graph
|
||||
else:
|
||||
node = random.choice(list(self.graph.nodes))
|
||||
if node not in sub_graph_nodes:
|
||||
sub_graph_nodes.append(node)
|
||||
|
||||
# Create the sub-graph
|
||||
sub_G = nx.DiGraph()
|
||||
sub_G.add_nodes_from(sub_graph_nodes)
|
||||
sub_G.add_edges_from(edges)
|
||||
|
||||
return sub_G
|
||||
|
||||
def sample_subgraph_random_walk_with_restart(self, seed_node, num_nodes, restart_prob=0.15):
|
||||
# Create a list to store the sub-graph nodes
|
||||
sub_graph_nodes = [seed_node]
|
||||
edges = []
|
||||
|
||||
# Keep adding nodes until we reach the desired number
|
||||
while len(sub_graph_nodes) < num_nodes:
|
||||
# Randomly select a node from the current sub-graph
|
||||
node = random.choice(sub_graph_nodes)
|
||||
neighbors = list(self.graph.successors(node))
|
||||
|
||||
# If the node has neighbors, randomly select one and add it to the sub-graph
|
||||
if neighbors:
|
||||
neighbor = random.choice(neighbors)
|
||||
if neighbor not in sub_graph_nodes:
|
||||
edges.append((node, neighbor))
|
||||
sub_graph_nodes.append(neighbor)
|
||||
# If the node has no neighbors, select a new node from the original graph
|
||||
else:
|
||||
node = random.choice(list(self.graph.nodes))
|
||||
if node not in sub_graph_nodes:
|
||||
sub_graph_nodes.append(node)
|
||||
|
||||
# Randomly restart the walk
|
||||
if random.random() < restart_prob:
|
||||
node = random.choice(list(self.graph.nodes))
|
||||
if node not in sub_graph_nodes:
|
||||
sub_graph_nodes.append(node)
|
||||
|
||||
# Create the sub-graph
|
||||
sub_G = nx.DiGraph()
|
||||
sub_G.add_nodes_from(sub_graph_nodes)
|
||||
sub_G.add_edges_from(edges)
|
||||
|
||||
return sub_G
|
||||
|
||||
@click.command()
|
||||
@click.option('--file_name', default='graph_desc_original.json', help='Path to the json file')
|
||||
@click.option('--sample_method', default='chain', help='Type of graph to generate')
|
||||
@click.option('--num_nodes', default=3, help='Number of nodes in the subgraph')
|
||||
@click.option('--save_figure', default=False, help='Save the figure')
|
||||
def sample_subgraph(file_name, sample_method, num_nodes, save_figure):
|
||||
# Create a graph sampler
|
||||
random.seed(0)
|
||||
sampler = GraphSampler(file_name=file_name)
|
||||
|
||||
# Sample a sub-graph
|
||||
sub_G = sampler.sample_subgraph(num_nodes, sample_method=sample_method)
|
||||
print("Sub-graph nodes:", sub_G.nodes)
|
||||
print("Sub-graph edges:", sub_G.edges)
|
||||
|
||||
# Visualize the sub-graph
|
||||
if save_figure:
|
||||
pos = nx.circular_layout(sub_G)
|
||||
nx.draw_networkx_nodes(sub_G, pos, node_color="skyblue", node_size=300)
|
||||
nx.draw_networkx_edges(sub_G, pos, arrows=True)
|
||||
nx.draw_networkx_labels(sub_G, pos, font_size=8)
|
||||
plt.axis("off")
|
||||
plt.tight_layout()
|
||||
plt.savefig("test.png")
|
||||
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sample_subgraph()
|
||||
@@ -0,0 +1,296 @@
|
||||
|
||||
import os
|
||||
import json
|
||||
import click
|
||||
import asyncio
|
||||
import aiohttp
|
||||
import logging
|
||||
import emoji
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
logger.setLevel(logging.INFO)
|
||||
logger.handlers = []
|
||||
|
||||
class RateLimitError(Exception):
|
||||
def __init__(self, message):
|
||||
super().__init__(message)
|
||||
|
||||
class ContentFormatError(Exception):
|
||||
def __init__(self, message):
|
||||
super().__init__(message)
|
||||
|
||||
@click.command()
|
||||
@click.option("--data_dir", default="data_huggingface", help="The directory of the data.")
|
||||
@click.option("--temperature", type=float, default=0.2)
|
||||
@click.option("--top_p", type=float, default=0.1)
|
||||
@click.option("--api_addr", type=str, default="localhost")
|
||||
@click.option("--api_port", type=int, default=4000)
|
||||
@click.option("--api_key", type=str, default="your api key")
|
||||
@click.option("--multiworker", type=int, default=1)
|
||||
@click.option("--llm", type=str, default="gpt-4")
|
||||
@click.option("--use_demos", type=int, default=0)
|
||||
@click.option("--reformat", type=bool, default=False)
|
||||
@click.option("--reformat_by", type=str, default="self")
|
||||
@click.option("--tag", type=bool, default=False)
|
||||
@click.option("--dependency_type", type=str, default="resource")
|
||||
@click.option("--log_first_detail", type=bool, default=False)
|
||||
def main(data_dir, temperature, top_p, api_addr, api_key, api_port, multiworker, llm, use_demos, reformat, reformat_by, tag, dependency_type, log_first_detail):
|
||||
assert dependency_type in ["resource", "temporal"], "Dependency type not supported"
|
||||
if dependency_type == "resource":
|
||||
assert data_dir != "data_dailylifeapis", "Resource dependency type only support data_huggingface and data_multimedia"
|
||||
|
||||
arguments = locals()
|
||||
url = f"http://{api_addr}:{api_port}/v1/chat/completions"
|
||||
header = {
|
||||
"Authorization": f"Bearer {api_key}",
|
||||
"Content-Type": "application/json"
|
||||
}
|
||||
|
||||
prediction_dir = f"{data_dir}/predictions{f'_use_demos_{use_demos}' if use_demos and tag else ''}{f'_reformat_by_{ reformat_by}' if reformat and tag else ''}"
|
||||
wf_name = f"{prediction_dir}/{llm}.json"
|
||||
|
||||
if not os.path.exists(prediction_dir):
|
||||
os.makedirs(prediction_dir, exist_ok=True)
|
||||
|
||||
has_inferenced = []
|
||||
if os.path.exists(wf_name):
|
||||
rf = open(wf_name, "r")
|
||||
for line in rf:
|
||||
data = json.loads(line)
|
||||
has_inferenced.append(data["id"])
|
||||
rf.close()
|
||||
|
||||
rf_ur = open(f"{data_dir}/user_requests.json", "r")
|
||||
inputs = []
|
||||
for line in rf_ur:
|
||||
input = json.loads(line)
|
||||
if input["id"] not in has_inferenced:
|
||||
inputs.append(input)
|
||||
rf_ur.close()
|
||||
|
||||
wf = open(wf_name, "a")
|
||||
|
||||
tool_list = json.load(open(f"{data_dir}/tool_desc.json", "r"))["nodes"]
|
||||
if "input-type" not in tool_list[0]:
|
||||
assert dependency_type == "temporal", "Tool type is not ignored, but the tool list does not contain input-type and output-type"
|
||||
if dependency_type == "temporal":
|
||||
for tool in tool_list:
|
||||
parameter_list = []
|
||||
for parameter in tool["parameters"]:
|
||||
parameter_list.append(parameter["name"])
|
||||
tool["parameters"] = parameter_list
|
||||
|
||||
# log llm name in format
|
||||
formatter = logging.Formatter(f"%(asctime)s - [ {llm} ] - %(levelname)s - %(message)s")
|
||||
console_handler = logging.StreamHandler()
|
||||
console_handler.setFormatter(formatter)
|
||||
console_handler.setLevel(logging.INFO)
|
||||
logger.addHandler(console_handler)
|
||||
|
||||
file_handler = logging.FileHandler(f"{prediction_dir}/{llm}.log")
|
||||
file_handler.setFormatter(formatter)
|
||||
file_handler.setLevel(logging.INFO)
|
||||
logger.addHandler(file_handler)
|
||||
|
||||
# logging all args
|
||||
logger.info(f"Arguments: {arguments}")
|
||||
|
||||
demos = []
|
||||
if use_demos:
|
||||
if dependency_type == "temporal":
|
||||
demos_id = [ "38563456", "27267145", "91005535"]
|
||||
else:
|
||||
if "huggingface" in data_dir:
|
||||
demos_id = [ "10523150", "14611002", "22067492"]
|
||||
elif "multimedia" in data_dir:
|
||||
demos_id = [ "30934207", "20566230", "19003517"]
|
||||
demos_id = demos_id[:use_demos]
|
||||
logger.info(f"Use {len(demos_id)} demos: {demos_id}")
|
||||
demos_rf = open(f"{data_dir}/data.json", "r")
|
||||
for line in demos_rf:
|
||||
data = json.loads(line)
|
||||
if data["id"] in demos_id:
|
||||
if dependency_type == "temporal":
|
||||
demo = {
|
||||
"user_request": data["user_request"],
|
||||
"result":{
|
||||
"task_steps": data["task_steps"],
|
||||
"task_nodes": data["task_nodes"],
|
||||
"task_links": data["task_links"]
|
||||
}
|
||||
}
|
||||
else:
|
||||
demo = {
|
||||
"user_request": data["user_request"],
|
||||
"result":{
|
||||
"task_steps": data["task_steps"],
|
||||
"task_nodes": data["task_nodes"]
|
||||
}
|
||||
}
|
||||
demos.append(demo)
|
||||
demos_rf.close()
|
||||
|
||||
tool_string = "# TASK LIST #:\n"
|
||||
for k, tool in enumerate(tool_list):
|
||||
tool_string += json.dumps(tool) + "\n"
|
||||
|
||||
sem = asyncio.Semaphore(multiworker)
|
||||
|
||||
async def inference_wrapper(input, url, header, temperature, top_p, tool_string, wf, llm, demos, reformat, reformat_by, dependency_type, log_detail = False):
|
||||
async with sem:
|
||||
await inference(input, url, header, temperature, top_p, tool_string, wf, llm, demos, reformat, reformat_by, dependency_type, log_detail)
|
||||
|
||||
if len(inputs) == 0:
|
||||
logger.info("All Completed!")
|
||||
return
|
||||
else:
|
||||
logger.info(f"Detected {len(has_inferenced)} has been inferenced,")
|
||||
logger.info(f"Start inferencing {len(inputs)} tasks...")
|
||||
|
||||
loop = asyncio.get_event_loop()
|
||||
|
||||
if log_first_detail:
|
||||
tasks = [inference_wrapper(inputs[0], url, header, temperature, top_p, tool_string, wf, llm, demos, reformat, reformat_by, dependency_type, log_detail=True)]
|
||||
results = loop.run_until_complete(asyncio.gather(*tasks, return_exceptions=True))
|
||||
inputs = inputs[1:]
|
||||
|
||||
tasks = []
|
||||
for input in inputs:
|
||||
tasks.append(inference_wrapper(input, url, header, temperature, top_p, tool_string, wf, llm, demos, reformat, reformat_by, dependency_type))
|
||||
|
||||
results += loop.run_until_complete(asyncio.gather(*tasks, return_exceptions=True))
|
||||
failed = []
|
||||
done = []
|
||||
for result in results:
|
||||
if isinstance(result, Exception):
|
||||
failed.append(result)
|
||||
else:
|
||||
done.append(result)
|
||||
logger.info(f"Completed: {len(done)}")
|
||||
logger.info(f"Failed: {len(failed)}")
|
||||
loop.close()
|
||||
|
||||
async def inference(input, url, header, temperature, top_p, tool_string, wf, llm, demos, reformat, reformat_by, dependency_type, log_detail = False):
|
||||
user_request = input["user_request"]
|
||||
if dependency_type == "resource":
|
||||
prompt = """\n# GOAL #: Based on the above tools, I want you generate task steps and task nodes to solve the # USER REQUEST #. The format must in a strict JSON format, like: {"task_steps": [ step description of one or more steps ], "task_nodes": [{"task": "tool name must be from # TOOL LIST #", "arguments": [ a concise list of arguments for the tool. Either original text, or user-mentioned filename, or tag '<node-j>' (start from 0) to refer to the output of the j-th node. ]}]} """
|
||||
prompt += """\n\n# REQUIREMENTS #: \n1. the generated task steps and task nodes can resolve the given user request # USER REQUEST # perfectly. Task name must be selected from # TASK LIST #; \n2. the task steps should strictly aligned with the task nodes, and the number of task steps should be same with the task nodes; \n3. the dependencies among task steps should align with the argument dependencies of the task nodes; \n4. the tool arguments should be align with the input-type field of # TASK LIST #;"""
|
||||
else:
|
||||
prompt = """\n# GOAL #:\nBased on the above tools, I want you generate task steps and task nodes to solve the # USER REQUEST #. The format must in a strict JSON format, like: {"task_steps": [ "concrete steps, format as Step x: Call xxx tool with xxx: 'xxx' and xxx: 'xxx'" ], "task_nodes": [{"task": "task name must be from # TASK LIST #", "arguments": [ {"name": "parameter name", "value": "parameter value, either user-specified text or the specific name of the tool whose result is required by this node"} ]}], "task_links": [{"source": "task name i", "target": "task name j"}]}"""
|
||||
prompt += """\n\n# REQUIREMENTS #: \n1. the generated task steps and task nodes can resolve the given user request # USER REQUEST # perfectly. Task name must be selected from # TASK LIST #; \n2. the task steps should strictly aligned with the task nodes, and the number of task steps should be same with the task nodes; \n3. The task links (task_links) should reflect the temporal dependencies among task nodes, i.e. the order in which the APIs are invoked;"""
|
||||
|
||||
if len(demos) > 0:
|
||||
prompt += "\n"
|
||||
for demo in demos:
|
||||
prompt += f"""\n# EXAMPLE #:\n# USER REQUEST #: {demo["user_request"]}\n# RESULT #: {json.dumps(demo["result"])}"""
|
||||
|
||||
prompt += """\n\n# USER REQUEST #: {{user_request}}\nnow please generate your result in a strict JSON format:\n# RESULT #:"""
|
||||
|
||||
final_prompt = tool_string + prompt.replace("{{user_request}}", user_request)
|
||||
payload = json.dumps({
|
||||
"model": f"{llm}",
|
||||
"messages": [
|
||||
{
|
||||
"role": "user",
|
||||
"content": final_prompt
|
||||
}
|
||||
],
|
||||
"temperature": temperature,
|
||||
"top_p": top_p,
|
||||
"frequency_penalty": 0,
|
||||
"presence_penalty": 1.05,
|
||||
"max_tokens": 2000,
|
||||
"stream": False,
|
||||
"stop": None
|
||||
})
|
||||
try:
|
||||
result = await get_response(url, header, payload, input['id'], reformat, reformat_by, dependency_type, log_detail)
|
||||
except Exception as e:
|
||||
logger.info(f"Failed #id {input['id']}: {type(e)} {e}")
|
||||
raise e
|
||||
logger.info(f"Success #id {input['id']}")
|
||||
input["result"] = result
|
||||
wf.write(json.dumps(input) + "\n")
|
||||
wf.flush()
|
||||
|
||||
async def get_response(url, header, payload, id, reformat, reformat_by, dependency_type, log_detail=False):
|
||||
async with aiohttp.ClientSession() as session:
|
||||
async with session.post(url, headers=header, data=payload, timeout=300) as response:
|
||||
resp = await response.json()
|
||||
|
||||
if response.status == 429:
|
||||
raise RateLimitError(f"{resp}")
|
||||
if response.status != 200:
|
||||
raise Exception(f"{resp}")
|
||||
|
||||
if log_detail:
|
||||
logger.info(json.loads(payload)["messages"][0]["content"])
|
||||
logger.info(resp["choices"][0]["message"]["content"])
|
||||
|
||||
oring_content = resp["choices"][0]["message"]["content"]
|
||||
oring_content = oring_content.replace("\n", "")
|
||||
oring_content = oring_content.replace("\_", "_")
|
||||
content = oring_content.replace("\\", "")
|
||||
|
||||
start_pos = content.find("RESULT #:")
|
||||
if start_pos!=-1:
|
||||
content = content[start_pos+len("RESULT #:"):]
|
||||
|
||||
content = content[content.find("{"):content.rfind("}")+1]
|
||||
try:
|
||||
content = json.loads(content)
|
||||
if isinstance(content, list) and len(content):
|
||||
merge_content = {}
|
||||
for c in content:
|
||||
for k, v in c.items():
|
||||
merge_content[k].extend(v) if k in merge_content else merge_content.update({k: v})
|
||||
return content
|
||||
except json.JSONDecodeError as e:
|
||||
if reformat:
|
||||
if dependency_type == "resource":
|
||||
prompt = """Please format the result # RESULT # to a strict JSON format # STRICT JSON FORMAT #. \nRequirements:\n1. Do not change the meaning of task steps and task nodes;\n2. Don't tolerate any possible irregular formatting to ensure that the generated content can be converted by json.loads();\n3. You must output the result in this schema: {"task_steps": [ step description of one or more steps ], "task_nodes": [{"task": "tool name must be from # TOOL LIST #", "arguments": [ a concise list of arguments for the tool. Either original text, or user-mentioned filename, or tag '<node-j>' (start from 0) to refer to the output of the j-th node. ]}]}\n# RESULT #:{{illegal_result}}\n# STRICT JSON FORMAT #:"""
|
||||
else:
|
||||
prompt = """Please format the result # RESULT # to a strict JSON format # STRICT JSON FORMAT #. \nRequirements:\n1. Do not change the meaning of task steps, task nodes and task links;\n2. Don't tolerate any possible irregular formatting to ensure that the generated content can be converted by json.loads();\n3. Pay attention to the matching of brackets. Write in a compact format and avoid using too many space formatting controls;\n4. You must output the result in this schema: {"task_steps": [ "concrete steps, format as Step x: Call xxx tool with xxx: 'xxx' and xxx: 'xxx'" ], "task_nodes": [{"task": "task name must be from # TASK LIST #", "arguments": [ {"name": "parameter name", "value": "parameter value, either user-specified text or the specific name of the tool whose result is required by this node"} ]}], "task_links": [{"source": "task name i", "target": "task name j"}]}\n# RESULT #:{{illegal_result}}\n# STRICT JSON FORMAT #:"""
|
||||
prompt = prompt.replace("{{illegal_result}}", oring_content)
|
||||
payload = json.loads(payload)
|
||||
if reformat_by != "self":
|
||||
payload["model"] = reformat_by
|
||||
|
||||
if log_detail:
|
||||
logger.info(f"{emoji.emojize(':warning:')} #id {id} Illegal JSON format: {content}")
|
||||
logger.info(f"{emoji.emojize(':sparkles:')} #id {id} Detected illegal JSON format, try to reformat by {payload['model']}...")
|
||||
|
||||
payload["messages"][0]["content"] = prompt
|
||||
payload = json.dumps(payload)
|
||||
|
||||
async with aiohttp.ClientSession() as session:
|
||||
async with session.post(url, headers=header, data=payload, timeout=120) as response:
|
||||
resp = await response.json()
|
||||
|
||||
if response.status == 429:
|
||||
raise RateLimitError(f"{resp}")
|
||||
if response.status != 200:
|
||||
raise Exception(f"{resp}")
|
||||
|
||||
if log_detail:
|
||||
logger.info(json.loads(payload)["messages"][0]["content"])
|
||||
logger.info(resp["choices"][0]["message"]["content"])
|
||||
|
||||
content = resp["choices"][0]["message"]["content"]
|
||||
content = content.replace("\n", "")
|
||||
content = content.replace("\_", "_")
|
||||
start_pos = content.find("STRICT JSON FORMAT #:")
|
||||
if start_pos!=-1:
|
||||
content = content[start_pos+len("STRICT JSON FORMAT #:"):]
|
||||
|
||||
content = content[content.find("{"):content.rfind("}")+1]
|
||||
try:
|
||||
content = json.loads(content)
|
||||
return content
|
||||
except json.JSONDecodeError as e:
|
||||
raise ContentFormatError(f"{content}")
|
||||
else:
|
||||
raise ContentFormatError(f"{content}")
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,12 @@
|
||||
requests==2.28.1
|
||||
click==8.0.4
|
||||
emoji==2.8.0
|
||||
networkx==2.8.4
|
||||
aiohttp==3.8.1
|
||||
matplotlib==3.7.1
|
||||
pandas==1.2.4
|
||||
numpy==1.23.5
|
||||
scikit-learn==1.0.2
|
||||
Levenshtein==0.21.1
|
||||
scipy==1.10.0
|
||||
datasets==2.14.5
|
||||
@@ -0,0 +1,37 @@
|
||||
import json
|
||||
import networkx as nx
|
||||
import matplotlib.pyplot as plt
|
||||
import click
|
||||
|
||||
@click.command()
|
||||
@click.option('--data_dir')
|
||||
def visialize_graph(data_dir):
|
||||
graph_file = f"{data_dir}/graph_desc.json"
|
||||
with open(graph_file, "r") as f:
|
||||
data = json.load(f)
|
||||
|
||||
G = nx.DiGraph()
|
||||
|
||||
for node in data["nodes"]:
|
||||
G.add_node(node["id"])
|
||||
|
||||
for link in data["links"]:
|
||||
G.add_edge(link["source"], link["target"])
|
||||
|
||||
pos = nx.spring_layout(G)
|
||||
pos = nx.random_layout(G)
|
||||
pos = nx.kamada_kawai_layout(G)
|
||||
|
||||
# Show the visualization
|
||||
plt.figure(figsize=(60, 60), dpi=80)
|
||||
plt.tight_layout()
|
||||
plt.axis("off")
|
||||
plt.show()
|
||||
|
||||
nx.draw_networkx_nodes(G, pos, node_color="skyblue", node_size=1200)
|
||||
nx.draw_networkx_edges(G, pos, arrows=True, arrowsize=40)
|
||||
nx.draw_networkx_labels(G, pos, font_size=50, font_color="green", font_weight="bold")
|
||||
plt.savefig(graph_file.replace(".json", ".pdf"))
|
||||
|
||||
if __name__ == "__main__":
|
||||
visialize_graph()
|
||||
Reference in New Issue
Block a user