commit ea5096c87319104d4cec366f46af5a2be9102cf2 Author: wehub-resource-sync Date: Mon Jul 13 13:06:23 2026 +0800 chore: import upstream snapshot with attribution diff --git a/.env.template b/.env.template new file mode 100644 index 0000000..b273695 --- /dev/null +++ b/.env.template @@ -0,0 +1,11 @@ +# Required Github Tokens +GITHUB_AI_TOKEN= + +# Optional API Keys +OPENAI_API_KEY= +DEEPSEEK_API_KEY= +ANTHROPIC_API_KEY= +GEMINI_API_KEY= +HUGGINGFACE_API_KEY= +GROQ_API_KEY= +XAI_API_KEY= diff --git a/.gitattributes b/.gitattributes new file mode 100644 index 0000000..eb761c8 --- /dev/null +++ b/.gitattributes @@ -0,0 +1 @@ +# *.mp4 filter=lfs diff=lfs merge=lfs -text diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..82348fb --- /dev/null +++ b/.gitignore @@ -0,0 +1,31 @@ +workplace_*/ +workspace_*/ +*.log +code_db/* +results/* + +__pycache__/ +tmp/* +logs/* +*.tar.gz + +*.egg-info + +.DS_Store +*.csv + +eval_data/* +evaluation_results/* +casestudy_results/* + +evaluation/*/data/ +evaluation/*/data/* +evaluation/**/data/ + +.env + +terminal_tmp/* + +!tool_docs.csv + +.port* diff --git a/Communication.md b/Communication.md new file mode 100644 index 0000000..00b5c7e --- /dev/null +++ b/Communication.md @@ -0,0 +1,6 @@ +We provide QR codes for joining the HKUDS discussion groups on WeChat and Feishu. + +You can join by scanning the QR codes below: + +WeChat QR Code + diff --git a/LICENSE b/LICENSE new file mode 100644 index 0000000..af1393b --- /dev/null +++ b/LICENSE @@ -0,0 +1,25 @@ +The MIT License (MIT) +===================== + +Copyright © 2023 + +Permission is hereby granted, free of charge, to any person +obtaining a copy of this software and associated documentation +files (the “Software”), to deal in the Software without +restriction, including without limitation the rights to use, +copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the +Software is furnished to do so, subject to the following +conditions: + +The above copyright notice and this permission notice shall be +included in all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, +EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES +OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND +NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT +HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, +WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING +FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR +OTHER DEALINGS IN THE SOFTWARE. diff --git a/README.md b/README.md new file mode 100644 index 0000000..735657a --- /dev/null +++ b/README.md @@ -0,0 +1,527 @@ + + +
+ Logo +

AutoAgent: Fully-Automated & Zero-Code
LLM Agent Framework

+
+ + + + +
+ Credits + Join our Slack community + Join our Discord community + + + + +
+ Check out the documentation + Paper + Evaluation Benchmark Score +
+
+ +
+HKUDS%2FAutoAgent | Trendshift +
+ +Welcome to AutoAgent! AutoAgent is a **Fully-Automated** and highly **Self-Developing** framework that enables users to create and deploy LLM agents through **Natural Language Alone**. + +## ✨Key Features of AutoAgent + +* 💬 **Natural Language-Driven Agent Building** +
Automatically constructs and orchestrates collaborative agent systems purely through natural dialogue, eliminating the need for manual coding or technical configuration. + +* 🚀 **Zero-Code Framework** +
Democratizes AI development by allowing anyone, regardless of coding experience, to create and customize their own agents, tools, and workflows using natural language alone. + +* ⚡ **Self-Managing Workflow Generation** +
Dynamically creates, optimizes and adapts agent workflows based on high-level task descriptions, even when users cannot fully specify implementation details. + +* 🔧 **Intelligent Resource Orchestration** +
Enables controlled code generation for creating tools, agents, and workflows through iterative self-improvement, supporting both single agent creation and multi-agent workflow generation. + +* 🎯 **Self-Play Agent Customization** +
Enables controlled code generation for creating tools, agents, and workflows through iterative self-improvement, supporting both single agent creation and multi-agent workflow generation. + +🚀 Unlock the Future of LLM Agents. Try 🔥AutoAgent🔥 Now! + +
+ +
+ Logo +
Quick Overview of AutoAgent.
+
+
+ + + +## 🔥 News + +
+ +
+ + +## 📑 Table of Contents + +* ✨ Features +* 🔥 News +* 🔍 How to Use AutoAgent + * 1. `user mode` (Deep Research Agents) + * 2. `agent editor` (Agent Creation without Workflow) + * 3. `workflow editor` (Agent Creation with Workflow) +* ⚡ Quick Start + * Installation + * API Keys Setup + * Start with CLI Mode +* ☑️ Todo List +* 🔬 How To Reproduce the Results in the Paper +* 📖 Documentation +* 🤝 Join the Community +* 🙏 Acknowledgements +* 🌟 Cite + + + +## 🔍 How to Use AutoAgent + + + +### 1. `user mode` (Deep Research Agents) + +AutoAgent features a ready-to-use multi-agent system accessible through user mode on the start page. This system serves as a comprehensive AI research assistant designed for information retrieval, complex analytical tasks, and comprehensive report generation. + +- 🚀 **High Performance**: Matches Deep Research using Claude 3.5 rather than OpenAI's o3 model. +- 🔄 **Model Flexibility**: Compatible with any LLM (including Deepseek-R1, Grok, Gemini, etc.) +- 💰 **Cost-Effective**: Open-source alternative to Deep Research's $200/month subscription +- 🎯 **User-Friendly**: Easy-to-deploy CLI interface for seamless interaction +- 📁 **File Support**: Handles file uploads for enhanced data interaction + +
+ +

🎥 Deep Research (aka User Mode)

+
+ + + + + +### 2. `agent editor` (Agent Creation without Workflow) + +The most distinctive feature of AutoAgent is its natural language customization capability. Unlike other agent frameworks, AutoAgent allows you to create tools, agents, and workflows using natural language alone. Simply choose `agent editor` or `workflow editor` mode to start your journey of building agents through conversations. + +You can use `agent editor` as shown in the following figure. + + + + + + + +
+ requirement +
+ Input what kind of agent you want to create. + 		+ profiling +
+ Automated agent profiling. + 		+ profiles +
+ Output the agent profiles. +
+ + + + + + +
+ tools +
+ Create the desired tools. + 		+ task +
+ Input what do you want to complete with the agent. (Optional) + 		+ output +
+ Create the desired agent(s) and go to the next step. +
+ + + +### 3. `workflow editor` (Agent Creation with Workflow) + +You can also create the agent workflows using natural language description with the `workflow editor` mode, as shown in the following figure. (Tips: this mode does not support tool creation temporarily.) + + + + + + + +
+ requirement +
+ Input what kind of workflow you want to create. + 		+ profiling +
+ Automated workflow profiling. + 		+ profiles +
+ Output the workflow profiles. +
+ + + + + +
+ task +
+ Input what do you want to complete with the workflow. (Optional) + 		+ output +
+ Create the desired workflow(s) and go to the next step. +
+ + + +## ⚡ Quick Start + + + +### Installation + +#### AutoAgent Installation + +```bash +git clone https://github.com/HKUDS/AutoAgent.git +cd AutoAgent +pip install -e . +``` + +#### Docker Installation + +We use Docker to containerize the agent-interactive environment. So please install [Docker](https://www.docker.com/) first. You don't need to manually pull the pre-built image, because we have let Auto-Deep-Research **automatically pull the pre-built image based on your architecture of your machine**. + + + +### API Keys Setup + +Create an environment variable file, just like `.env.template`, and set the API keys for the LLMs you want to use. Not every LLM API Key is required, use what you need. + +```bash +# Required Github Tokens of your own +GITHUB_AI_TOKEN= + +# Optional API Keys +OPENAI_API_KEY= +DEEPSEEK_API_KEY= +ANTHROPIC_API_KEY= +GEMINI_API_KEY= +HUGGINGFACE_API_KEY= +GROQ_API_KEY= +XAI_API_KEY= +``` + + + +### Start with CLI Mode + +> [🚨 **News**: ] We have updated a more easy-to-use command to start the CLI mode and fix the bug of different LLM providers from issues. You can follow the following steps to start the CLI mode with different LLM providers with much less configuration. + +#### Command Options: + +You can run `auto main` to start full part of AutoAgent, including `user mode`, `agent editor` and `workflow editor`. Btw, you can also run `auto deep-research` to start more lightweight `user mode`, just like the [Auto-Deep-Research](https://github.com/HKUDS/Auto-Deep-Research) project. Some configuration of this command is shown below. + +- `--container_name`: Name of the Docker container (default: 'deepresearch') +- `--port`: Port for the container (default: 12346) +- `COMPLETION_MODEL`: Specify the LLM model to use, you should follow the name of [Litellm](https://github.com/BerriAI/litellm) to set the model name. (Default: `claude-3-5-sonnet-20241022`) +- `DEBUG`: Enable debug mode for detailed logs (default: False) +- `API_BASE_URL`: The base URL for the LLM provider (default: None) +- `FN_CALL`: Enable function calling (default: None). Most of time, you could ignore this option because we have already set the default value based on the model name. +- `git_clone`: Clone the AutoAgent repository to the local environment (only support with the `auto main` command, default: True) +- `test_pull_name`: The name of the test pull. (only support with the `auto main` command, default: 'autoagent_mirror') + +#### More details about `git_clone` and `test_pull_name`] + +In the `agent editor` and `workflow editor` mode, we should clone a mirror of the AutoAgent repository to the local agent-interactive environment and let our **AutoAgent** automatically update the AutoAgent itself, such as creating new tools, agents and workflows. So if you want to use the `agent editor` and `workflow editor` mode, you should set the `git_clone` to True and set the `test_pull_name` to 'autoagent_mirror' or other branches. + +#### `auto main` with different LLM Providers + +Then I will show you how to use the full part of AutoAgent with the `auto main` command and different LLM providers. If you want to use the `auto deep-research` command, you can refer to the [Auto-Deep-Research](https://github.com/HKUDS/Auto-Deep-Research) project for more details. + +##### Anthropic + +* set the `ANTHROPIC_API_KEY` in the `.env` file. + +```bash +ANTHROPIC_API_KEY=your_anthropic_api_key +``` + +* run the following command to start Auto-Deep-Research. + +```bash +auto main # default model is claude-3-5-sonnet-20241022 +``` + +##### OpenAI + +* set the `OPENAI_API_KEY` in the `.env` file. + +```bash +OPENAI_API_KEY=your_openai_api_key +``` + +* run the following command to start Auto-Deep-Research. + +```bash +COMPLETION_MODEL=gpt-4o auto main +``` + +##### Mistral + +* set the `MISTRAL_API_KEY` in the `.env` file. + +```bash +MISTRAL_API_KEY=your_mistral_api_key +``` + +* run the following command to start Auto-Deep-Research. + +```bash +COMPLETION_MODEL=mistral/mistral-large-2407 auto main +``` + +##### Gemini - Google AI Studio + +* set the `GEMINI_API_KEY` in the `.env` file. + +```bash +GEMINI_API_KEY=your_gemini_api_key +``` + +* run the following command to start Auto-Deep-Research. + +```bash +COMPLETION_MODEL=gemini/gemini-2.0-flash auto main +``` + +##### Huggingface + +* set the `HUGGINGFACE_API_KEY` in the `.env` file. + +```bash +HUGGINGFACE_API_KEY=your_huggingface_api_key +``` + +* run the following command to start Auto-Deep-Research. + +```bash +COMPLETION_MODEL=huggingface/meta-llama/Llama-3.3-70B-Instruct auto main +``` + +##### Groq + +* set the `GROQ_API_KEY` in the `.env` file. + +```bash +GROQ_API_KEY=your_groq_api_key +``` + +* run the following command to start Auto-Deep-Research. + +```bash +COMPLETION_MODEL=groq/deepseek-r1-distill-llama-70b auto main +``` + +##### OpenAI-Compatible Endpoints (e.g., Grok) + +* set the `OPENAI_API_KEY` in the `.env` file. + +```bash +OPENAI_API_KEY=your_api_key_for_openai_compatible_endpoints +``` + +* run the following command to start Auto-Deep-Research. + +```bash +COMPLETION_MODEL=openai/grok-2-latest API_BASE_URL=https://api.x.ai/v1 auto main +``` + +##### OpenRouter (e.g., DeepSeek-R1) + +We recommend using OpenRouter as LLM provider of DeepSeek-R1 temporarily. Because official API of DeepSeek-R1 can not be used efficiently. + +* set the `OPENROUTER_API_KEY` in the `.env` file. + +```bash +OPENROUTER_API_KEY=your_openrouter_api_key +``` + +* run the following command to start Auto-Deep-Research. + +```bash +COMPLETION_MODEL=openrouter/deepseek/deepseek-r1 auto main +``` + +##### DeepSeek + +* set the `DEEPSEEK_API_KEY` in the `.env` file. + +```bash +DEEPSEEK_API_KEY=your_deepseek_api_key +``` + +* run the following command to start Auto-Deep-Research. + +```bash +COMPLETION_MODEL=deepseek/deepseek-chat auto main +``` + + +After the CLI mode is started, you can see the start page of AutoAgent: + +
+ +
+ Logo +
Start Page of AutoAgent.
+
+
+ +### Tips + +#### Import browser cookies to browser environment + +You can import the browser cookies to the browser environment to let the agent better access some specific websites. For more details, please refer to the [cookies](./AutoAgent/environment/cookie_json/README.md) folder. + +#### Add your own API keys for third-party Tool Platforms + +If you want to create tools from the third-party tool platforms, such as RapidAPI, you should subscribe tools from the platform and add your own API keys by running [process_tool_docs.py](./process_tool_docs.py). + +```bash +python process_tool_docs.py +``` + +More features coming soon! 🚀 **Web GUI interface** under development. + + + + + +## ☑️ Todo List + +AutoAgent is continuously evolving! Here's what's coming: + +- 📊 **More Benchmarks**: Expanding evaluations to **SWE-bench**, **WebArena**, and more +- 🖥️ **GUI Agent**: Supporting *Computer-Use* agents with GUI interaction +- 🔧 **Tool Platforms**: Integration with more platforms like **Composio** +- 🏗️ **Code Sandboxes**: Supporting additional environments like **E2B** +- 🎨 **Web Interface**: Developing comprehensive GUI for better user experience + +Have ideas or suggestions? Feel free to open an issue! Stay tuned for more exciting updates! 🚀 + + + +## 🔬 How To Reproduce the Results in the Paper + +### GAIA Benchmark +For the GAIA benchmark, you can run the following command to run the inference. + +```bash +cd path/to/AutoAgent && sh evaluation/gaia/scripts/run_infer.sh +``` + +For the evaluation, you can run the following command. + +```bash +cd path/to/AutoAgent && python evaluation/gaia/get_score.py +``` + +### Agentic-RAG + +For the Agentic-RAG task, you can run the following command to run the inference. + +Step1. Turn to [this page](https://huggingface.co/datasets/yixuantt/MultiHopRAG) and download it. Save them to your datapath. + +Step2. Run the following command to run the inference. + +```bash +cd path/to/AutoAgent && sh evaluation/multihoprag/scripts/run_rag.sh +``` + +Step3. The result will be saved in the `evaluation/multihoprag/result.json`. + + + +## 📖 Documentation + +A more detailed documentation is coming soon 🚀, and we will update in the [Documentation](https://AutoAgent-ai.github.io/docs) page. + + + +## 🤝 Join the Community + +We want to build a community for AutoAgent, and we welcome everyone to join us. You can join our community by: + +- [Join our Slack workspace](https://join.slack.com/t/AutoAgent-workspace/shared_invite/zt-2zibtmutw-v7xOJObBf9jE2w3x7nctFQ) - Here we talk about research, architecture, and future development. +- [Join our Discord server](https://discord.gg/z68KRvwB) - This is a community-run server for general discussion, questions, and feedback. +- [Read or post Github Issues](https://github.com/HKUDS/AutoAgent/issues) - Check out the issues we're working on, or add your own ideas. + + + + + +## Misc + +
+ +[![Stargazers repo roster for @HKUDS/AutoAgent](https://reporoster.com/stars/HKUDS/AutoAgent)](https://github.com/HKUDS/AutoAgent/stargazers) + +[![Forkers repo roster for @HKUDS/AutoAgent](https://reporoster.com/forks/HKUDS/AutoAgent)](https://github.com/HKUDS/AutoAgent/network/members) + +[![Star History Chart](https://api.star-history.com/svg?repos=HKUDS/AutoAgent&type=Date)](https://star-history.com/#HKUDS/AutoAgent&Date) + +
+ +## 🙏 Acknowledgements + +Rome wasn't built in a day. AutoAgent stands on the shoulders of giants, and we are deeply grateful for the outstanding work that came before us. Our framework architecture draws inspiration from [OpenAI Swarm](https://github.com/openai/swarm), while our user mode's three-agent design benefits from [Magentic-one](https://github.com/microsoft/autogen/tree/main/python/packages/autogen-magentic-one)'s insights. We've also learned from [OpenHands](https://github.com/All-Hands-AI/OpenHands) for documentation structure and many other excellent projects for agent-environment interaction design, among others. We express our sincere gratitude and respect to all these pioneering works that have been instrumental in shaping AutoAgent. + + + + +## 🌟 Cite + +```tex +@misc{AutoAgent, + title={{AutoAgent: A Fully-Automated and Zero-Code Framework for LLM Agents}}, + author={Jiabin Tang, Tianyu Fan, Chao Huang}, + year={2025}, + eprint={202502.05957}, + archivePrefix={arXiv}, + primaryClass={cs.AI}, + url={https://arxiv.org/abs/2502.05957}, +} +``` + + + + + diff --git a/README.wehub.md b/README.wehub.md new file mode 100644 index 0000000..508c15d --- /dev/null +++ b/README.wehub.md @@ -0,0 +1,7 @@ +# WeHub 来源说明 + +- 原始项目：`HKUDS/AutoAgent` +- 原始仓库：https://github.com/HKUDS/AutoAgent +- 导入方式：上游默认分支的最新快照 +- 原作者、版权和许可证信息以原始仓库及本仓库 LICENSE 为准 +- 本文件仅用于记录来源，不代表 WeHub 是原项目作者 diff --git a/assets/AutoAgent_Arxiv_2025.pdf b/assets/AutoAgent_Arxiv_2025.pdf new file mode 100644 index 0000000..f2ce097 Binary files /dev/null and b/assets/AutoAgent_Arxiv_2025.pdf differ diff --git a/assets/AutoAgent_logo.svg b/assets/AutoAgent_logo.svg new file mode 100644 index 0000000..6a2ddc7 --- /dev/null +++ b/assets/AutoAgent_logo.svg @@ -0,0 +1,4 @@ + + + + \ No newline at end of file diff --git a/assets/QR.png b/assets/QR.png new file mode 100644 index 0000000..b5add69 Binary files /dev/null and b/assets/QR.png differ diff --git a/assets/agent_editor/1-requirement.png b/assets/agent_editor/1-requirement.png new file mode 100644 index 0000000..397de98 Binary files /dev/null and b/assets/agent_editor/1-requirement.png differ diff --git a/assets/agent_editor/2-profiling.png b/assets/agent_editor/2-profiling.png new file mode 100644 index 0000000..1c86b13 Binary files /dev/null and b/assets/agent_editor/2-profiling.png differ diff --git a/assets/agent_editor/3-profiles.png b/assets/agent_editor/3-profiles.png new file mode 100644 index 0000000..dad2b26 Binary files /dev/null and b/assets/agent_editor/3-profiles.png differ diff --git a/assets/agent_editor/4-tools.png b/assets/agent_editor/4-tools.png new file mode 100644 index 0000000..8c7fbaf Binary files /dev/null and b/assets/agent_editor/4-tools.png differ diff --git a/assets/agent_editor/5-task.png b/assets/agent_editor/5-task.png new file mode 100644 index 0000000..9363f74 Binary files /dev/null and b/assets/agent_editor/5-task.png differ diff --git a/assets/agent_editor/6-output-next.png b/assets/agent_editor/6-output-next.png new file mode 100644 index 0000000..57357f5 Binary files /dev/null and b/assets/agent_editor/6-output-next.png differ diff --git a/assets/arrow.png b/assets/arrow.png new file mode 100644 index 0000000..5b41a12 Binary files /dev/null and b/assets/arrow.png differ diff --git a/assets/autoagent-intro-final.pdf b/assets/autoagent-intro-final.pdf new file mode 100644 index 0000000..3e525be Binary files /dev/null and b/assets/autoagent-intro-final.pdf differ diff --git a/assets/autoagent-intro.pdf b/assets/autoagent-intro.pdf new file mode 100644 index 0000000..fc175a8 Binary files /dev/null and b/assets/autoagent-intro.pdf differ diff --git a/assets/autoagent-intro.png b/assets/autoagent-intro.png new file mode 100644 index 0000000..dce5d0a Binary files /dev/null and b/assets/autoagent-intro.png differ diff --git a/assets/autoagent-intro.svg b/assets/autoagent-intro.svg new file mode 100644 index 0000000..fb0b0c2 --- /dev/null +++ b/assets/autoagent-intro.svg @@ -0,0 +1,4 @@ + + + +



Web
Agent
Coding
Agent

Browser

Terminal

Local File
Agent

Markdown


      Profiling
Agent
      Profiling
Agent
        Parsing
        Editor
Agent
 Create       /
        Editor
Agent
        Editor
Agent
Create       /
        Editor
Agent

LLM Providers

DB Providers

 Chroma

Tool Platform

Programming

Debugging

...

Veiwing

Analysing

...

Creating

Tools

...

Creating

Agents

...

Creating

Workflow

...

Local Files

Programs

Browser

Automated Generation of Customizable Agents, Tools, Workflows


User Interface

Self-Evolving Agents via Self-Play Customization

Generalist Agent Stack for Search·Code·Files

Fully-Automated and Zero-Code Agent Framework

Language-Driven Framework

Give me a comparative analysis report of AAPL and MSFT stocks.

Self-Developing

Browsing

Websites

...


Workflow Name:
majority_voting_flow
Workflow Description: 
System Input: ..
System Output: ..
Agents:[agent1, ..]
Workflow Code:
```python
import autoagent
..

Agent Name:
Financial Agent 
Agent Description: ..
Agent Instruction: ..
Agent Tools: 
[Tool_1, Tool_2, ..]
Agent Code:
```python
import autoagent
..

Tool Name: get_cash_flow
Tool Description: ..
Tool Argument: 
{id: .., period: ..}
Tool Code:
```python
import autoagent
```
I automatically generated all the code and created 
Financial Agent. I run the agent and get the analysis report: 
1. Portfolio Allocation Strategy: ...

 \ No newline at end of file diff --git a/assets/autoagent-wechat.jpg b/assets/autoagent-wechat.jpg new file mode 100644 index 0000000..24f8d8d Binary files /dev/null and b/assets/autoagent-wechat.jpg differ diff --git a/assets/autoagent.png b/assets/autoagent.png new file mode 100644 index 0000000..acf2366 Binary files /dev/null and b/assets/autoagent.png differ diff --git a/assets/cookies/export.png b/assets/cookies/export.png new file mode 100644 index 0000000..c2b9fb5 Binary files /dev/null and b/assets/cookies/export.png differ diff --git a/assets/cookies/extension.png b/assets/cookies/extension.png new file mode 100644 index 0000000..99bd50b Binary files /dev/null and b/assets/cookies/extension.png differ diff --git a/assets/cover.png b/assets/cover.png new file mode 100644 index 0000000..523fcdd Binary files /dev/null and b/assets/cover.png differ diff --git a/assets/paper/metachain_arxiv.pdf b/assets/paper/metachain_arxiv.pdf new file mode 100644 index 0000000..f89867e Binary files /dev/null and b/assets/paper/metachain_arxiv.pdf differ diff --git a/assets/rec.png b/assets/rec.png new file mode 100644 index 0000000..ae4308f Binary files /dev/null and b/assets/rec.png differ diff --git a/assets/technology-278512.mp3 b/assets/technology-278512.mp3 new file mode 100644 index 0000000..044a977 Binary files /dev/null and b/assets/technology-278512.mp3 differ diff --git a/assets/user_mode/4-upload.png b/assets/user_mode/4-upload.png new file mode 100644 index 0000000..bbd91c8 Binary files /dev/null and b/assets/user_mode/4-upload.png differ diff --git a/assets/user_mode/5-select.png b/assets/user_mode/5-select.png new file mode 100644 index 0000000..958c968 Binary files /dev/null and b/assets/user_mode/5-select.png differ diff --git a/assets/user_mode/6-success.png b/assets/user_mode/6-success.png new file mode 100644 index 0000000..2bd183b Binary files /dev/null and b/assets/user_mode/6-success.png differ diff --git a/assets/user_mode/input.png b/assets/user_mode/input.png new file mode 100644 index 0000000..8e1b913 Binary files /dev/null and b/assets/user_mode/input.png differ diff --git a/assets/user_mode/output.png b/assets/user_mode/output.png new file mode 100644 index 0000000..2d27f30 Binary files /dev/null and b/assets/user_mode/output.png differ diff --git a/assets/user_mode/select_agent.png b/assets/user_mode/select_agent.png new file mode 100644 index 0000000..5164cc2 Binary files /dev/null and b/assets/user_mode/select_agent.png differ diff --git a/assets/video_v1_compressed.mp4 b/assets/video_v1_compressed.mp4 new file mode 100644 index 0000000..aa9d95e Binary files /dev/null and b/assets/video_v1_compressed.mp4 differ diff --git a/assets/workflow_editor/1-requirement.png b/assets/workflow_editor/1-requirement.png new file mode 100644 index 0000000..b45c1df Binary files /dev/null and b/assets/workflow_editor/1-requirement.png differ diff --git a/assets/workflow_editor/2-profiling.png b/assets/workflow_editor/2-profiling.png new file mode 100644 index 0000000..9fbd407 Binary files /dev/null and b/assets/workflow_editor/2-profiling.png differ diff --git a/assets/workflow_editor/3-profiles.png b/assets/workflow_editor/3-profiles.png new file mode 100644 index 0000000..73f50e7 Binary files /dev/null and b/assets/workflow_editor/3-profiles.png differ diff --git a/assets/workflow_editor/4-task.png b/assets/workflow_editor/4-task.png new file mode 100644 index 0000000..3636f2e Binary files /dev/null and b/assets/workflow_editor/4-task.png differ diff --git a/assets/workflow_editor/5-output-next.png b/assets/workflow_editor/5-output-next.png new file mode 100644 index 0000000..1fb415f Binary files /dev/null and b/assets/workflow_editor/5-output-next.png differ diff --git a/autoagent/__init__.py b/autoagent/__init__.py new file mode 100644 index 0000000..047cf3b --- /dev/null +++ b/autoagent/__init__.py @@ -0,0 +1,9 @@ +from .core import MetaChain +from .types import Agent, Response +# from .workflow import Graph, meta_workflow, FlowEngine +from .flow import default_drive + +import autoagent.workflows +import autoagent.tools +import autoagent.agents +__all__ = ["MetaChain", "Agent", "Response", "default_drive", ] diff --git a/autoagent/agents/__init__.py b/autoagent/agents/__init__.py new file mode 100644 index 0000000..f30f420 --- /dev/null +++ b/autoagent/agents/__init__.py @@ -0,0 +1,64 @@ +# from autoagent.agents.programming_agent import get_programming_agent +# from autoagent.agents.tool_retriver_agent import get_tool_retriver_agent +# from autoagent.agents.agent_check_agent import get_agent_check_agent +# from autoagent.agents.tool_check_agent import get_tool_check_agent +# from autoagent.agents.github_agent import get_github_agent +# from autoagent.agents.programming_triage_agent import get_programming_triage_agent +# from autoagent.agents.plan_agent import get_plan_agent + +# import os +# import importlib +# from autoagent.registry import registry + +# # 获取当前目录下的所有 .py 文件 +# current_dir = os.path.dirname(__file__) +# for file in os.listdir(current_dir): +# if file.endswith('.py') and not file.startswith('__'): +# module_name = file[:-3] +# importlib.import_module(f'autoagent.agents.{module_name}') + +# # 导出所有注册的 agent 创建函数 +# globals().update(registry.agents) + +# __all__ = list(registry.agents.keys()) + +import os +import importlib +from autoagent.registry import registry + +def import_agents_recursively(base_dir: str, base_package: str): + """Recursively import all agents in .py files + + Args: + base_dir: the root directory to start searching + base_package: the base name of the Python package + """ + for root, dirs, files in os.walk(base_dir): + # get the relative path to the base directory + rel_path = os.path.relpath(root, base_dir) + + for file in files: + if file.endswith('.py') and not file.startswith('__'): + # build the module path + if rel_path == '.': + # in the root directory + module_path = f"{base_package}.{file[:-3]}" + else: + # in the subdirectory + package_path = rel_path.replace(os.path.sep, '.') + module_path = f"{base_package}.{package_path}.{file[:-3]}" + + try: + importlib.import_module(module_path) + except Exception as e: + print(f"Warning: Failed to import {module_path}: {e}") + +# get the current directory and import all agents +current_dir = os.path.dirname(__file__) +import_agents_recursively(current_dir, 'autoagent.agents') + +# export all agent creation functions +globals().update(registry.agents) +globals().update(registry.plugin_agents) + +__all__ = list(registry.agents.keys()) \ No newline at end of file diff --git a/autoagent/agents/dummy_agent.py b/autoagent/agents/dummy_agent.py new file mode 100644 index 0000000..6840e51 --- /dev/null +++ b/autoagent/agents/dummy_agent.py @@ -0,0 +1,36 @@ +from autoagent.types import Agent +from autoagent.tools import tool_dummy +from typing import Union +from autoagent.registry import register_plugin_agent # import the register_agent function from the registry + +@register_plugin_agent(name = "Dummy Agent", func_name="get_dummy_agent") # You must register the agent in the registry, otherwise the agent will not be loaded. The name of register_agent is get_xxx_agent. +def get_dummy_agent(model: str): + """ + This is a dummy agent, it's used for demonstrating the usage of the autoagent. + Args: + model: The model to be used for the agent. + Returns: + An agent instance. + """ + def dummy_instructions(context_variables: dict): + """ + The function should take the context_variables as an argument, and return a string. The context_variables is a dictionary, and it's track the important variables of the agent in the whole conversation. + The instructions should be concise and clear, and it's very important for the agent to follow the instructions. + """ + tmp_variables = context_variables.get("tmp_variables", {}) + return f"""...""" + return Agent( + name="Dummy Agent", # The name of the agent, you can change it in different scenes. + model=model, # The default model is gpt-4o-2024-08-06, you can change it to other models if user specified. + instructions="..." or dummy_instructions, # the instructions of the agent, the instructions can be a string or a function that returns a string. If it is a function, the function should take the context_variables as an argument, and return a string. The instructions should be concise and clear, and it's very important for the agent to follow the instructions. + functions=[tool_dummy], # The tools of the agent, you can add different tools in different scenes. + ) + +""" +Form to create an agent: + +agent_name = "Dummy Agent" +agent_description = "This is a dummy agent, it's used for demonstrating the usage of the autoagent." +agent_instructions = "..." | "...{global_variables}..." +agent_tools = [tool_dummy] +""" \ No newline at end of file diff --git a/autoagent/agents/github_agent.py b/autoagent/agents/github_agent.py new file mode 100644 index 0000000..991ad86 --- /dev/null +++ b/autoagent/agents/github_agent.py @@ -0,0 +1,25 @@ +from autoagent.types import Agent +from autoagent.tools import ( + push_changes, submit_pull_request +) +from autoagent.registry import register_agent + + +@register_agent(name = "Github Agent", func_name="get_github_agent") +def get_github_agent(model: str): + def instructions(context_variables): + return \ +f"""You are an agent that helps user to manage the GitHub repository named 'autoagent'. +The user will give you the suggestion of the changes to be pushed to the repository. +Follow the following routine with the user: +1. First, use `push_changes` to push the changes to the repository. (If the user want to push all the changes, use `push_changes` with `file_paths=None` as the argument.) +2. Then, ask the user whether to submit a pull request to a target branch. (If yes, give the `target_branch`) +3. If the user wants to submit a pull request, use `submit_pull_request` to submit the pull request, if not, just ignore this step. +""" + return Agent( + name="Github Agent", + model=model, + instructions=instructions, + functions=[push_changes, submit_pull_request], + parallel_tool_calls = False + ) diff --git a/autoagent/agents/math/math_solver_agent.py b/autoagent/agents/math/math_solver_agent.py new file mode 100644 index 0000000..1e3ec57 --- /dev/null +++ b/autoagent/agents/math/math_solver_agent.py @@ -0,0 +1,17 @@ +from autoagent.types import Agent + +from autoagent.registry import register_plugin_agent + +@register_plugin_agent(name="Math Solver Agent", func_name="get_math_solver_agent") +def get_math_solver_agent(model: str): + ''' + This agent solves mathematical problems using analytical and systematic approaches. + ''' + instructions = 'You are responsible for solving mathematical problems using a systematic approach. You should:\n1. Use the provided conditions and objective to formulate a solution strategy\n2. Break down complex problems into smaller steps\n3. Apply appropriate mathematical concepts and formulas\n4. Show clear step-by-step work and explanations\n5. Verify the solution matches the problem requirements' + return Agent( + name="Math Solver Agent", + model=model, + instructions=instructions, + functions=[] + ) + diff --git a/autoagent/agents/math/vote_aggregator_agent.py b/autoagent/agents/math/vote_aggregator_agent.py new file mode 100644 index 0000000..f88b223 --- /dev/null +++ b/autoagent/agents/math/vote_aggregator_agent.py @@ -0,0 +1,17 @@ +from autoagent.types import Agent + +from autoagent.registry import register_plugin_agent + +@register_plugin_agent(name="Vote Aggregator Agent", func_name="get_vote_aggregator_agent") +def get_vote_aggregator_agent(model: str): + ''' + This agent aggregates solutions from different solvers and determines the final answer through majority voting. + ''' + instructions = 'You are a solution aggregator specializing in combining and analyzing multiple solutions to determine the most accurate answer. Your responsibilities include:\n\n1. Carefully review all provided solutions\n2. Compare the reasoning and calculations in each solution\n3. Identify commonalities and differences between solutions\n4. Implement majority voting when solutions differ\n5. Evaluate the confidence level of each solution\n6. Provide justification for the final selected answer\n\nWhen aggregating solutions:\n1. List all solutions received\n2. Compare the approach and methodology used in each\n3. Identify the final answer from each solution\n4. Apply majority voting to determine the consensus\n5. If no clear majority, analyze the reasoning quality to break ties\n6. Present the final selected answer with explanation of the selection process' + return Agent( + name="Vote Aggregator Agent", + model=model, + instructions=instructions, + functions=[] + ) + diff --git a/autoagent/agents/meta_agent/agent_creator.py b/autoagent/agents/meta_agent/agent_creator.py new file mode 100644 index 0000000..5637bbb --- /dev/null +++ b/autoagent/agents/meta_agent/agent_creator.py @@ -0,0 +1,78 @@ +from autoagent.registry import register_agent +from autoagent.tools.meta.edit_agents import list_agents, create_agent, delete_agent, run_agent, read_agent, create_orchestrator_agent +from autoagent.tools.meta.edit_tools import list_tools, create_tool, delete_tool, run_tool +from autoagent.tools.terminal_tools import execute_command, terminal_page_down, terminal_page_up, terminal_page_to +from autoagent.types import Agent +from autoagent.io_utils import read_file + + +@register_agent(name = "Agent Creator Agent", func_name="get_agent_creator_agent") +def get_agent_creator_agent(model: str) -> str: + """ + The agent creator is an agent that can be used to create the agents. + """ + def instructions(context_variables): + return f"""\ +You are an Agent Creator specialized in the MetaChain framework. Your primary responsibility is to create, manage, and orchestrate agents based on XML-formatted agent forms. + +CORE RESPONSIBILITIES: +1. Parse and implement agent forms +2. Create and manage individual agents +3. Orchestrate multi-agent systems +4. Handle dependencies and system requirements + +AVAILABLE FUNCTIONS: +1. Agent Management: + - `create_agent`: Create new agents or update existing ones strictly following the given agent form. + - `read_agent`: Retrieve existing agent definitions. Note that if you want to use `create_agent` to update an existing agent, you MUST use the `read_agent` function to get the definition of the agent first. + - `delete_agent`: Remove unnecessary agents. + - `list_agents`: Display all available agents and their information. + - `create_orchestrator_agent`: Create orchestrator for multi-agent systems. If the request is to create MORE THAN ONE agent, after you create ALL required agents, you MUST use the `create_orchestrator_agent` function to create an orchestrator agent that can orchestrate the workflow of the agents. And then use the `run_agent` function to run the orchestrator agent to complete the user task. + +2. Execution: + - run_agent: Execute agent to complete the user task. The agent could be a single agent (single agent form) or an orchestrator agent (multi-agent form). + - execute_command: Handle system dependencies and requirements + - terminal_page_down: Move the terminal page down when the terminal output is too long. + - terminal_page_up: Move the terminal page up when the terminal output is too long. + - terminal_page_to: Move the terminal page to the specific page when the terminal output is too long, and you want to move to the specific page with the meaningful content. + +WORKFLOW GUIDELINES: + +1. Single Agent Implementation: + - Carefully read the agent form and understand the requirements. + - Create/update agent using create_agent + - Execute task using run_agent + - Monitor and handle any errors + +2. Multi-Agent Implementation: + - Create all required agents individually using `create_agent` + - MUST create an orchestrator agent using `create_orchestrator_agent` + - Execute task through the `run_agent` function to execute the created orchestrator agent + - Monitor system performance + +3. Error Handling: + - Check for missing dependencies using `execute_command` + - Install required packages using execute_command + - Validate agent creation and execution + - Report any issues clearly + +BEST PRACTICES: +1. Always verify existing agents using `read_agent` before updates +2. Create orchestrator agents for ANY multi-agent scenario using `create_orchestrator_agent` +3. Handle dependencies proactively using `execute_command` +4. Maintain clear documentation of created agents +5. Follow the exact specifications from the agent form XML + +Remember: Your success is measured by both the accurate creation of agents and their effective execution of the given tasks. +""" + tool_list = [list_agents, create_agent, delete_agent, run_agent, execute_command, read_agent, create_orchestrator_agent, terminal_page_down, terminal_page_up, terminal_page_to] + return Agent( + name="Agent Creator Agent", + model=model, + instructions=instructions, + functions=tool_list, + tool_choice = "required", + parallel_tool_calls = False + ) + + diff --git a/autoagent/agents/meta_agent/agent_editor.py b/autoagent/agents/meta_agent/agent_editor.py new file mode 100644 index 0000000..cd1ae03 --- /dev/null +++ b/autoagent/agents/meta_agent/agent_editor.py @@ -0,0 +1,38 @@ +from autoagent.registry import register_agent +from autoagent.tools.meta.edit_agents import list_agents, create_agent, delete_agent, run_agent +from autoagent.tools.terminal_tools import execute_command +from autoagent.types import Agent +from autoagent.io_utils import read_file + +@register_agent(name = "Agent Editor Agent", func_name="get_agent_editor_agent") +def get_agent_editor_agent(model: str) -> str: + """ + The agent editor is an agent that can be used to edit the agents. + """ + def instructions(context_variables): + return f"""\ +You are an agent editor agent that can be used to edit the agents. You are working on a Agent framework named MetaChain, and your responsibility is to edit the agents in the MetaChain, so that the agents can be used to help the user with their request. + +The existing agents are shown below: +{list_agents(context_variables)} + +If you want to create a new agent, you should: +1. follow the format of the `get_dummy_agent` below: +```python +{read_file('autoagent/agents/dummy_agent.py')} +``` +2. you successfully create the agent only after you have successfully run the agent with the `run_agent` function to satisfy the user's request. + +3. If you encounter any error while creating and running the agent, like dependency missing, you should use the `execute_command` function to install the dependency. + +[IMPORTANT] The `register_plugin_agent` registry function is strictly required for a agent implementation to be recognized by the MetaChain framework. +""" + tool_list = [list_agents, create_agent, delete_agent, run_agent, execute_command] + return Agent( + name="Agent Editor Agent", + model=model, + instructions=instructions, + functions=tool_list, + tool_choice = "required", + parallel_tool_calls = False + ) diff --git a/autoagent/agents/meta_agent/agent_former.py b/autoagent/agents/meta_agent/agent_former.py new file mode 100644 index 0000000..7b89dce --- /dev/null +++ b/autoagent/agents/meta_agent/agent_former.py @@ -0,0 +1,257 @@ +from autoagent.registry import register_agent +from autoagent.tools.meta.edit_agents import list_agents, create_agent, delete_agent, run_agent, read_agent +from autoagent.tools.meta.edit_tools import list_tools, create_tool, delete_tool, run_tool +from autoagent.tools.terminal_tools import execute_command +from autoagent.types import Agent +from autoagent.io_utils import read_file +from pydantic import BaseModel, Field +from typing import List + + +@register_agent(name = "Agent Former Agent", func_name="get_agent_former_agent") +def get_agent_former_agent(model: str) -> str: + """ + This agent is used to complete a form that can be used to create an agent. + """ + def instructions(context_variables): + return r"""\ +You are an agent specialized in creating agent forms for the MetaChain framework. + +Your task is to analyze user requests and generate structured creation forms for either single or multi-agent systems. + +KEY COMPONENTS OF THE FORM: +1. - Root element containing all agent definitions + +2. - Defines what the system receives + - Must describe the overall input that the system accepts + - For single agent: Same as agent_input + - For multi-agent: Should encompass all possible inputs that will be routed to different agents + +3. - Specifies system response format + - Must contain exactly ONE key-description pair + - : Single identifier for the system's output + - : Explanation of the output + - For single agent: Same as agent_output + - For multi-agent: Should represent the unified output format from all agents + +4. - Individual agent definition + - name: Agent's identifier + - description: Agent's purpose and capabilities + - instructions: Agent's behavioral guidelines + * To reference global variables, use format syntax: {variable_key} + * Example: "Help the user {user_name} with his/her request" + * All referenced keys must exist in global_variables + - tools: Available tools (existing/new) + - agent_input: + * Must contain exactly ONE key-description pair + * : Identifier for the input this agent accepts + * : Detailed explanation of the input format + - agent_output: + * Must contain exactly ONE key-description pair + * : Identifier for what this agent produces + * : Detailed explanation of the output format + +5. - Shared variables across agents (optional) + - Used for constants or shared values accessible by all agents + - Variables defined here can be referenced in instructions using {key} + - Example: + ```xml + + + user_name + The name of the user + John Doe + + + ``` + - Usage in instructions: "You are a personal assistant for {user_name}." + +IMPORTANT RULES: +- For single agent systems: + * system_input/output must match agent_input/output exactly +- For multi-agent systems: + * system_input should describe the complete input space + * Each agent_input should specify which subset of the system_input it handles + * system_output should represent the unified response format +""" + \ +f""" +Existing tools you can use is: +{list_tools(context_variables)} + +Existing agents you can use is: +{list_agents(context_variables)} +""" + \ +r""" +EXAMPLE 1 - SINGLE AGENT: + +User: I want to build an agent that can answer the user's question about the OpenAI products. The document of the OpenAI products is available at `/workspace/docs/openai_products/`. +The agent should be able to: +1. query and answer the user's question about the OpenAI products based on the document. +2. send email to the user if the sending email is required in the user's request. + +The form should be: + + + Questions from the user about the OpenAI products. The document of the OpenAI products is available at `/workspace/docs/openai_products/`. + + + answer + The answer to the user's question. + + + Helper Center Agent + The helper center agent is an agent that serves as a helper center agent for a specific user to answer the user's question about the OpenAI products. + You are a helper center agent that can be used to help the user with their request. + + + save_raw_docs_to_vector_db + Save the raw documents to the vector database. The documents could be: + - ANY text document with the extension of pdf, docx, txt, etcs. + - A zip file containing multiple text documents + - a directory containing multiple text documents + All documents will be converted to raw text format and saved to the vector database in the chunks of 4096 tokens. + + + query_db + Query the vector database to find the answer to the user's question. + + + modify_query + Modify the user's question to a more specific question. + + + answer_query + Answer the user's question based on the answer from the vector database. + + + can_answer + Check if the user's question can be answered by the vector database. + + + + + send_email + Send an email to the user. + + + + user_question + The question from the user about the OpenAI products. + + + answer + The answer to the user's question. + + + + +EXAMPLE 2 - MULTI-AGENT: + +User: I want to build a multi-agent system that can handle two types of requests for the specific user: +1. Purchase a product or service +2. Refund a product or service +The specific user worked for is named John Doe. + +The form should be: + + + The user request from the specific user about the product or service, mainly categorized into 2 types: + - Purchase a product or service + - Refund a product or service + + + response + The response of the agent to the user's request. + + + + user_name + The name of the user. + John Doe + + + + Personal Sales Agent + The personal sales agent is an agent that serves as a personal sales agent for a specific user. + You are a personal sales agent that can be used to help the user {user_name} with their request. + + + recommend_product + Recommend a product to the user. + + + recommend_service + Recommend a service to the user. + + + conduct_sales + Conduct sales with the user. + + + + user_request + Request from the specific user for purchasing a product or service. + + + response + The response of the agent to the user's request. + + + + Personal Refunds Agent + The personal refunds agent is an agent that serves as a personal refunds agent for a specific user. + Help the user {user_name} with a refund. If the reason is that it was too expensive, offer the user a discount. If they insist, then process the refund. + + + process_refund + Refund an item. Refund an item. Make sure you have the item_id of the form item_... Ask for user confirmation before processing the refund. + + + apply_discount + Apply a discount to the user's cart. + + + + user_request + Request from the specific user for refunding a product or service. + + + response + The response of the agent to the user's request. + + + + +GUIDELINES: +1. Each agent must have clear, focused responsibilities +2. Tool selections should be minimal but sufficient +3. Instructions should be specific and actionable +4. Input/Output definitions must be precise +5. Use global_variables for shared context across agents + +Follow these examples and guidelines to create appropriate agent forms based on user requirements. +""" + return Agent( + name = "Agent Former Agent", + model = model, + instructions = instructions, + ) + +if __name__ == "__main__": + from autoagent import MetaChain + agent = get_agent_former_agent("claude-3-5-sonnet-20241022") + client = MetaChain() + task_yaml = """\ +I want to create two agents that can help me to do two kinds of tasks: +1. Manage the private financial docs. I have a folder called `financial_docs` in my local machine, and I want to help me to manage the financial docs. +2. Search the financial information online. You may help me to: +- get balance sheets for a given ticker over a given period. +- get cash flow statements for a given ticker over a given period. +- get income statements for a given ticker over a given period. +""" + task_yaml = task_yaml + """\ +Directly output the form in the XML format. +""" + messages = [{"role": "user", "content": task_yaml}] + response = client.run(agent, messages) + print(response.messages[-1]["content"]) \ No newline at end of file diff --git a/autoagent/agents/meta_agent/form_complie.py b/autoagent/agents/meta_agent/form_complie.py new file mode 100644 index 0000000..9fe1104 --- /dev/null +++ b/autoagent/agents/meta_agent/form_complie.py @@ -0,0 +1,139 @@ +from pydantic import BaseModel, Field, validator, field_validator, ValidationInfo +from typing import List, Dict, Optional, Literal +import xml.etree.ElementTree as ET + +class KeyDescription(BaseModel): + key: str + description: str + +class Tool(BaseModel): + name: str + description: str + +class ToolSet(BaseModel): + existing: List[Tool] = Field(default_factory=list) + new: List[Tool] = Field(default_factory=list) + +class GlobalVariable(BaseModel): + key: str + description: str + value: str + +class Agent(BaseModel): + name: str + description: str + instructions: str + tools: ToolSet + agent_input: KeyDescription + agent_output: KeyDescription + +class AgentForm(BaseModel): + system_input: str + system_output: KeyDescription + global_variables: Dict[str, GlobalVariable] = Field(default_factory=dict) + agents: List[Agent] + + @field_validator('agents') + def validate_single_agent_io(cls, v, info: ValidationInfo): + """验证单agent系统的输入输出是否匹配""" + if len(v) == 1: + agent = v[0] + system_output = info.data.get('system_output') + if system_output and agent.agent_output.key != system_output.key: + raise ValueError("Single agent system must have matching system and agent output keys") + return v + # def validate_global_ctx_instructions(cls, v, info: ValidationInfo): + # """验证全局变量和系统输入是否匹配""" + +class XMLParser: + @staticmethod + def parse_key_description(elem: ET.Element, tag_name: str) -> KeyDescription: + node = elem.find(tag_name) + if node is None: + raise ValueError(f"Missing {tag_name}") + return KeyDescription( + key=node.find('key').text.strip(), + description=node.find('description').text.strip() + ) + + @staticmethod + def parse_tools(agent_elem: ET.Element) -> ToolSet: + tools = ToolSet() + for tools_elem in agent_elem.findall('tools'): + category = tools_elem.get('category') + if category not in ('existing', 'new'): + continue + + tool_list = [] + for tool_elem in tools_elem.findall('tool'): + tool = Tool( + name=tool_elem.find('name').text.strip(), + description=tool_elem.find('description').text.strip() + ) + tool_list.append(tool) + + if category == 'existing': + tools.existing = tool_list + else: + tools.new = tool_list + + return tools + + @staticmethod + def parse_global_variables(root: ET.Element) -> Dict[str, GlobalVariable]: + variables = {} + global_vars = root.find('global_variables') + if global_vars is not None: + for var in global_vars.findall('variable'): + key = var.find('key').text.strip() + variables[key] = GlobalVariable( + key=key, + description=var.find('description').text.strip(), + value=var.find('value').text.strip() + ) + return variables + + @classmethod + def parse_agent(cls, agent_elem: ET.Element) -> Agent: + return Agent( + name=agent_elem.find('name').text.strip(), + description=agent_elem.find('description').text.strip(), + instructions=agent_elem.find('instructions').text.strip(), + tools=cls.parse_tools(agent_elem), + agent_input=cls.parse_key_description(agent_elem, 'agent_input'), + agent_output=cls.parse_key_description(agent_elem, 'agent_output') + ) + + @classmethod + def parse_xml(cls, xml_content: str) -> AgentForm: + root = ET.fromstring(xml_content) + + return AgentForm( + system_input=root.find('system_input').text.strip(), + system_output=cls.parse_key_description(root, 'system_output'), + global_variables=cls.parse_global_variables(root), + agents=[cls.parse_agent(agent_elem) for agent_elem in root.findall('agent')] + ) + +def parse_agent_form(xml_content: str) -> Optional[AgentForm]: + """ + 读取并解析agent form XML文件 + + Args: + xml_content: XML文件内容 + + Returns: + 解析后的AgentForm对象，如果解析失败返回None + """ + try: + # with open(xml_path, 'r', encoding='utf-8') as f: + # xml_content = f.read() + + return XMLParser.parse_xml(xml_content) + + except ET.ParseError as e: + print(f"Error parsing XML: {e}") + return None + except Exception as e: + print(f"Unexpected error: {e}") + return None diff --git a/autoagent/agents/meta_agent/tool_editor.py b/autoagent/agents/meta_agent/tool_editor.py new file mode 100644 index 0000000..3df6412 --- /dev/null +++ b/autoagent/agents/meta_agent/tool_editor.py @@ -0,0 +1,257 @@ +from autoagent.registry import register_agent +from autoagent.tools.meta.edit_tools import list_tools, create_tool, delete_tool, run_tool, get_metachain_path +from autoagent.tools.meta.tool_retriever import get_api_plugin_tools_doc +from autoagent.tools.meta.search_tools import search_trending_models_on_huggingface, get_hf_model_tools_doc + +from autoagent.types import Agent +from autoagent.io_utils import read_file +from autoagent.tools.terminal_tools import execute_command, terminal_page_down, terminal_page_up, terminal_page_to +@register_agent(name = "Tool Editor Agent", func_name="get_tool_editor_agent") +def get_tool_editor_agent(model: str) -> Agent: + """ + The tool editor is an agent that can be used to edit the tools. + """ + def instructions(context_variables): + return f"""\ +You are a tool editor agent responsible for managing plugin tools in the MetaChain framework. Your core responsibility is to edit, create, and manage plugin tools that can be used by other agents. + +[PLUGIN TOOLS SYSTEM] +- Plugin tools are the building blocks of MetaChain +- All available plugin tools are as follows: +{list_tools(context_variables)} +- Plugin tools can ONLY be executed using `run_tool(tool_name, run_code)`. You should import `run_tool` by `from autoagent.tools import run_tool`. +- NEVER try to import and run plugin tools directly - always use `run_tool` + +[TOOL CREATION WORKFLOW] +1. ALWAYS start with `list_tools()` to check existing tools + +2. For NEW plugin tool creation, FOLLOW THIS ORDER: + a. For third-party API integration (e.g., RapidAPI, external services): + - MUST FIRST use `get_api_plugin_tools_doc` to get API documentation and keys + - API keys should be embedded IN the function body, NOT as parameters. + - The API keys are always in the retrieved information from `get_api_plugin_tools_doc`, DO NOT guess the API keys by yourself. + - Follow the API implementation details from the documentation + + b. For modal transformation tasks (image/video/audio generation/processing): + - FIRST use `search_trending_models_on_huggingface` to find suitable models, only support the following tags: ['audio-text-to-text', 'text-to-image', 'image-to-image', 'image-to-video', 'text-to-video', 'text-to-speech', 'text-to-audio', 'automatic-speech-recognition', 'audio-to-audio']. + - Then use `get_hf_model_tools_doc` for detailed model information + - Only use internal knowledge if no suitable models are found + + c. For visual analysis tasks (images/videos): + - MUST use the existing `visual_question_answering` plugin tool by `run_tool("visual_question_answering", "from autoagent.tools import visual_question_answering; ...")`. DO NOT use it directly without `run_tool`. + - NO direct implementation of visual processing + - Chain with other tools as needed + +3. Plugin Tool Implementation Requirements: + - Use @register_plugin_tool decorator (REQUIRED). You should import `register_plugin_tool` by `from autoagent.registry import register_plugin_tool`. + - Follow this template: +```python +{read_file('autoagent/tools/dummy_tool.py')} +``` + - Include clear type hints + - Make tools abstract and reusable + - Use generic names (e.g., 'process_media' not 'process_youtube_video') + - Handle dependencies with `execute_command` + +[AVAILABLE TOOLS] +1. get_api_plugin_tools_doc: + - PRIMARY tool for third-party API integration + - MUST be used FIRST for Finance, Entertainment, eCommerce, etc. + - Provides API documentation AND authentication keys + - API keys should be embedded in tool implementation + +2. search_trending_models_on_huggingface: + - Use for finding models for media transformation tasks + - Supported tags: ['text-to-image', 'image-to-image', 'text-to-video', etc.] + - Use AFTER checking no suitable API exists via `get_api_plugin_tools_doc` + +3. get_hf_model_tools_doc: + - Get the detailed information of a model on Hugging Face, such as the detailed usage of the model containing the model's README.md. + - You should use this tool after you have used `search_trending_models_on_huggingface` to find the model you want to use. + +4. Other management tools: + - list_tools(): Check existing tools + - create_tool(tool_name, tool_code): Create new tools + - run_tool(tool_name, run_code): REQUIRED method to execute any plugin tool + - delete_tool(tool_name): Remove tools + - execute_command: Install dependencies. Handles system-level operations + - terminal_page_* tools: Navigate long outputs + +5. case_resolved & case_not_resolved: + - case_resolved: after you have created all the tools and tested them using `run_tool` successfully (with the expected output rather than just run it), you should use the `case_resolved` tool to brief the result. + - case_not_resolved: after you have tried your best to create the tools but failed, you should use the `case_not_resolved` tool to tell the failure reason. + +[CRITICAL RULES] +1. Tool Creation Priority: + - FIRST: Check existing tools via list_tools() + - SECOND: Use `get_api_plugin_tools_doc` for API-based tools + - THIRD: Use `search_trending_models_on_huggingface` for media tasks + - LAST: Use internal knowledge if no other options available + +2. API Implementation: + - NEVER expose API keys as parameters + - ALWAYS embed API keys in function body + - Get keys from `get_api_plugin_tools_doc` + +3. Tool Design: + - Tools MUST be abstract, modular, and reusable: + - Use generic function names (e.g., `download_media` instead of `download_youtube_video`) + - Break complex tasks into smaller, reusable components + - Avoid task-specific implementations + - Use parameters instead of hardcoded values + - Include proper error handling + +[TESTING] +Test new tools using `run_tool`: +`run_tool(tool_name="your_tool", run_code="from autoagent.tools import your_tool; print(your_tool(param1='value1'))")` +""" + tool_list = [list_tools, create_tool, run_tool, delete_tool, get_api_plugin_tools_doc, execute_command, terminal_page_down, terminal_page_up, terminal_page_to, search_trending_models_on_huggingface, get_hf_model_tools_doc] + return Agent( + name="Tool Editor Agent", + model=model, + instructions=instructions, + functions=tool_list, + tool_choice = "required", + parallel_tool_calls = False + ) + + +""" +5. [IMPORTANT] If you want to use Hugging Face models, especially for some tasks related to vision, audio, video, you should use the `search_trending_models_on_huggingface` tool to search trending models related to the specific task on Hugging Face, and then use the `get_hf_model_tools_doc` tool to get the detailed information about the specific model. + +6. [IMPORTANT] As for the tags ['image-text-to-text', 'visual-question-answering', 'video-text-to-text'] and ANY visual tasks, you should use `visual_question_answering` tool instead of Hugging Face models. +""" + +"""\ +You are a tool editor agent that can be used to edit the tools. You are working on a Agent framework named MetaChain, and your responsibility is to edit the tools in the MetaChain, so that the tools can be used by the agents to help the user with their request. + +The existing tools are shown below: +{list_tools(context_variables)} + +If you want to create a new tool, you should: +1. follow the format of the `tool_dummy` below. Note that if the tool should be used with third-part api key, you should write the api key inside the definition of the tool: +```python +{read_file('autoagent/tools/dummy_tool.py')} +``` + +2. you successfully create the tool only after you have successfully run the tool with the `run_tool` function, and an example of testing the tool is shown below.: +```python +from autoagent.tools import tool_dummy + +if __name__ == "__main__": + ... # some pre-operations + print(run_tool(tool_name="tool_dummy", run_code="from autoagent.tools import tool_dummy; print(tool_dummy(args1=args1, args2=args1, ...))")) +``` + +3. If you encounter any error while creating and running the tool, like dependency missing, you should use the `execute_command` function to install the dependency. + +4. [IMPORTANT] If you want to use third-party api, especially for some tasks related to Finance, Entertainment, eCommerce, Food, Travel, Sports, you MUST use the `get_api_plugin_tools_doc` tool to search information from existing api documents, it contains how to implement the api and API keys. + +[IMPORTANT] The `register_plugin_tool` registry function is strictly required for a tool implementation to be recognized by the MetaChain framework. + +[IMPORTANT] The tool you create should be abstract, modular, and reusable. Specifically, the function name must be generic (e.g., +`count_objects` instead of `count_apples`). The function must use parameters instead of hard-coded values. The +function body must be self-contained. + +[IMPORTANT] Explicitly declare input and output data types using type hints. + +[IMPORTANT] For ANY visual tasks related to image and video, you should use `visual_question_answering` tool. +""" + + +"""\ +You are a tool editor agent responsible for managing plugin tools in the MetaChain framework. Your core responsibility is to edit, create, and manage plugin tools that can be used by other agents. + +[PLUGIN TOOLS SYSTEM] +- Plugin tools are the building blocks of MetaChain +- All available plugin tools are as follows: +{list_tools(context_variables)} +- Plugin tools can ONLY be executed using `run_tool(tool_name, run_code)` +- NEVER try to import and run tools directly - always use `run_tool` + +[AVAILABLE MANAGEMENT TOOLS] +1. list_tools(): + - Lists all existing plugin tools + - Returns: tool name, arguments, docstring, implementation details + - Use this FIRST to check existing tools + +2. create_tool(tool_name: str, tool_code: str): + - Creates new plugin tools + - Requires proper registration using @register_plugin_tool, and you MUST import `register_plugin_tool` by `from autoagent.registry import register_plugin_tool` + +3. run_tool(tool_name: str, run_code: str,): + - REQUIRED method to execute any plugin tool + - Format: run_tool("tool_name", "from autoagent.tools import tool_name; print(tool_name(args))") + +4. delete_tool(tool_name: str,): + - Removes existing plugin tools + - Use with caution + +5. get_api_plugin_tools_doc: + - Required for third-party API integrations, e.g. RapidAPI. + - MUST be used for Finance, Entertainment, etc. + +6. execute_command: + - Handles system-level operations + - Use for dependency installation + +7. terminal_page_down: + - Move the terminal page down when the terminal output is too long. + +8. terminal_page_up: + - Move the terminal page up when the terminal output is too long. + +9. terminal_page_to: + - Move the terminal page to the specific page when the terminal output is too long, and you want to move to the specific page with the meaningful content. + +10. search_trending_models_on_huggingface: + - Search trending models on Hugging Face. + - Use this tool when you want to use Hugging Face models to generate images, videos, audios, etc. + - Do NOT use this tool for text-to-text or image-to-text tasks. + +11. get_hf_model_tools_doc: + - Get the detailed information about the specific model on Hugging Face. + - Use this tool when you want to use Hugging Face models to generate images, videos, audios, etc. + +[CRITICAL PRINCIPLES FOR PLUGIN TOOLS] +1. Tools MUST be abstract, modular, and reusable: + - Use generic function names (e.g., `download_media` instead of `download_youtube_video`) + - Break complex tasks into smaller, reusable components + - Avoid task-specific implementations + - Use parameters instead of hardcoded values + +2. For ALL visual tasks (images, videos, visual analysis): + - MUST use the existing `visual_question_answering` plugin tool + - NO direct implementation of visual processing + - Chain `visual_question_answering` with other tools as needed + +[WORKFLOW FOR PLUGIN TOOL MANAGEMENT] +1. Always start with `list_tools()` to check existing tools +2. For new plugin tools: + a. Design generic, reusable interface + b. Follow the template format: +```python +{read_file('autoagent/tools/dummy_tool.py')} +``` + c. Create using `create_tool` + d. Test using `run_tool` + e. Handle dependencies with `execute_command` + +[IMPORTANT RULES] +- ALL tools must be registered with @register_plugin_tool +- ALL tools must have type hints +- Each tool does ONE thing well +- Create modular tools that can be combined +- ALWAYS use `run_tool` to execute plugin tools +- NEVER modify the `visual_question_answering` tool + +[TOOL TESTING EXAMPLE] +Correct way to test a plugin tool: +```python +result = run_tool( + tool_name="your_tool", + run_code="from autoagent.tools import your_tool; print(your_tool(param1='value1'))", + context_variables=context_variables +) +``` +""" \ No newline at end of file diff --git a/autoagent/agents/meta_agent/workflow_creator.py b/autoagent/agents/meta_agent/workflow_creator.py new file mode 100644 index 0000000..0e477ba --- /dev/null +++ b/autoagent/agents/meta_agent/workflow_creator.py @@ -0,0 +1,81 @@ +from autoagent.registry import register_agent +from autoagent.tools.meta.edit_agents import list_agents, create_agent, delete_agent, run_agent, read_agent, create_orchestrator_agent +from autoagent.tools.meta.edit_workflow import list_workflows, create_workflow, run_workflow +from autoagent.tools.terminal_tools import execute_command, terminal_page_down, terminal_page_up, terminal_page_to +from autoagent.types import Agent +from autoagent.io_utils import read_file + + +@register_agent(name = "Workflow Creator Agent", func_name="get_workflow_creator_agent") +def get_workflow_creator_agent(model: str) -> str: + """ + The workflow creator is an agent that can be used to create the workflow. + """ + def instructions(context_variables): + return f"""\ +You are a Workflow Creator specialized in the MetaChain framework. Your primary responsibility is to create and manage workflows based on XML-formatted workflow forms. + +CORE RESPONSIBILITIES: +1. Parse and implement workflow forms +2. Create necessary agents if specified in the workflow +3. Create and manage workflows +4. Execute workflows as needed + +AVAILABLE FUNCTIONS: +1. Workflow Management: + - `create_workflow`: Create new workflows based on the workflow form + - `run_workflow`: Execute the created workflow + - `list_workflows`: Display all available workflows + +2. Agent Management (when needed): + - `create_agent`: Create new agents if specified in the workflow form. If no tools are explicitly specified, use empty tool list ([]) + - `read_agent`: Retrieve existing agent definitions before updates + - `list_agents`: Display all available agents + +3. System Tools: + - `execute_command`: Handle system dependencies + - `terminal_page_down`, `terminal_page_up`, `terminal_page_to`: Navigate terminal output + +WORKFLOW CREATION PROCESS: + +1. Parse Workflow Form: + - Analyze the workflow form carefully + - Identify any new agents that need to be created + - Understand the workflow structure and requirements + +2. Create Required Agents: + - For each new agent in the workflow form: + * Use `create_agent` with appropriate parameters + * If no tools specified, use empty tool list ([]) + * Verify agent creation success + +3. Create Workflow: + - Use `create_workflow` to generate the workflow + - Ensure all required agents exist + - Validate workflow structure + +4. Execute Workflow: + - Use `run_workflow` to execute the created workflow + - Monitor execution progress + - Handle any errors appropriately + +BEST PRACTICES: +1. Always check if required agents exist before creating new ones +2. Use empty tool list ([]) when no specific tools are mentioned +3. Validate workflow creation before execution +4. Follow the exact specifications from the workflow form XML +5. Handle errors and dependencies appropriately + +Remember: Your primary goal is to create and execute workflows according to the provided workflow forms, creating any necessary agents along the way. +""" + tool_list = [list_agents, create_agent, execute_command, read_agent, terminal_page_down, terminal_page_up, terminal_page_to, list_workflows, create_workflow, run_workflow] + return Agent( + name="Workflow Creator Agent", + model=model, + instructions=instructions, + functions=tool_list, + tool_choice = "required", + parallel_tool_calls = False + ) + + diff --git a/autoagent/agents/meta_agent/workflow_former.py b/autoagent/agents/meta_agent/workflow_former.py new file mode 100644 index 0000000..afd438c --- /dev/null +++ b/autoagent/agents/meta_agent/workflow_former.py @@ -0,0 +1,578 @@ +from autoagent.registry import register_agent +from autoagent.tools.meta.edit_agents import list_agents, create_agent, delete_agent, run_agent, read_agent +from autoagent.tools.meta.edit_tools import list_tools, create_tool, delete_tool, run_tool +from autoagent.tools.meta.edit_workflow import list_workflows +from autoagent.tools.terminal_tools import execute_command +from autoagent.types import Agent +from autoagent.io_utils import read_file +from pydantic import BaseModel, Field +from typing import List +import json + + +@register_agent(name = "Workflow Former Agent", func_name="get_workflow_former_agent") +def get_workflow_former_agent(model: str) -> str: + """ + This agent is used to complete a form that can be used to create a workflow consisting of multiple agents. + """ + def instructions(context_variables): + workflow_list = list_workflows(context_variables) + workflow_list = json.loads(workflow_list) + workflow_list = [workflow_name for workflow_name in workflow_list.keys()] + workflow_list_str = ", ".join(workflow_list) + return r"""\ +You are an agent specialized in creating workflow forms for the MetaChain framework. + +Your task is to analyze user requests and generate structured creation forms for workflows consisting of multiple agents. + +KEY COMPONENTS OF THE FORM: +1. - Root element containing the entire workflow definition + +2. - The name of the workflow. It should be a single word with '_' as the separator, and as unique as possible to describe the speciality of the workflow. + +3. - Defines what the system receives + - Must describe the overall input that the system accepts + - : Single identifier for the input, could be a single word with '_' as the separator. + - : Detailed explanation of input format + +4. - Specifies system response format + - Must contain exactly ONE key-description pair + - : Single identifier for the system's output, could be a single word with '_' as the separator. + - : Explanation of the output format + + +5. - Contains all agent definitions + - Each can be existing or new (specified by category attribute) + - name: Agent's identifier + - description: Agent's purpose and capabilities + - tools: (optional): Only required for new agents when specific tools are requested + * Only include when user explicitly requests certain tools + +6. - Shared variables across agents in the workflow (optional) + - Used for constants or shared values accessible by all agents in EVERY event in the workflow + - Example: + ```xml + + + user_name + The name of the user + John Doe + + + ``` + +7. - Defines the workflow execution flow + Each contains: + - name: Event identifier + - inputs: What this event receives, should exactly match with the output keys of the events it's listening to + * Each input has: + - key: Input identifier (should match an output key from listened events) + - description: Input explanation + - task: What this event should accomplish + - outputs: Possible outcomes of this event + * Each output has: + - action: What happens after. Every action has a type and a optional value. Action is categorized into 3 types: + - RESULT: The event is successful, and the workflow will continue to the next event which is listening to this event. Value is the output of this event. + - ABORT: The event is not successful, and the workflow will abort. Value could be empty. + - GOTO: The event is not successful, and the workflow will wait for the next event. Value is the name of the event to go to. The event go to should NOT listen to this event. + - key: Output identifier (be a single word with '_' as the separator) + - description: Output explanation + - condition: when the output occurs, the action will be executed + * Can have single or multiple outputs: + - For single output (simple flow): + ```xml + + + result_key + Description of the result + + RESULT + + + + ``` + - For multiple outputs (conditional flow): + ```xml + + + success_result + Output when condition A is met + When condition A is true + + RESULT + + + + should_repeat + Output when condition B is met + When condition B is true + + GOTO + target_event + + + + failure_result + Output when condition C is met + When condition C is true + + ABORT + + + + ``` + - listen: Which events trigger this one. + - agent: Which agent handles this event. Every agent has the name of the agent, and the exact model of the agent (like `claude-3-5-sonnet-20241022` or others) + + +IMPORTANT RULES: +0. The `on_start` event is a special event that: + - Must be the first event in the workflow + - Has inputs that match the system_input + - Has outputs that match the system_input (just pass through) + - Does not have an agent + - Does not have a task + - Does not have listen elements + Example: + ```xml + + on_start + + + user_topic + The user's topic that user wants to write a wikipiead-like article about. + + + + + user_topic + The user's topic that user wants to write a wikipiead-like article about. + + RESULT + + + + + ``` + +1. For simple sequential flows: + - Use single output with RESULT type + - No condition is needed + - Next event in chain listening to this event will be triggered automatically + +2. For conditional flows: + - Multiple outputs must each have a condition + - Conditions should be mutually exclusive + - Each output should specify appropriate action type + - `GOTO` action should have a value which is the name of the event to go to + +3. Only include tools section when: + - Agent is new (category="new") AND + - User explicitly requests specific tools for the agent + +4. Omit tools section when: + - Using existing agents (category="existing") OR + - Creating new agents without specific tool requirements +""" + \ +f""" +Existing tools you can use is: +{list_tools(context_variables)} + +Existing agents you can use is: +{list_agents(context_variables)} + +The name of existing workflows: [{workflow_list_str}]. The name of the new workflow you are creating should be DIFFERENT from these names according to the speciality of the workflow. +""" + \ +r""" +COMMON WORKFLOW PATTERNS: + +1. If-Else Pattern (Conditional Branching): +```xml + + analyze_data + Analyze the data and determine next steps + + + positive_case + Handle positive case + If data meets criteria A + + RESULT + + + + negative_case + Handle the negative case + If data does not meet criteria A + + ABORT + + + + +``` + +2. Parallelization Pattern (Concurrent Execution): +```xml + + + initial_analysis + + + analysis_result + Initial analysis result + + RESULT + + + + + + + + technical_analysis + + initial_analysis + + + + technical_result + Technical analysis result + + RESULT + + + + + + + financial_analysis + + initial_analysis + + + + financial_result + Financial analysis result + + RESULT + + + + + + + + combine_results + + + technical_result + The technical analysis result. + + + financial_result + The financial analysis result. + + + + technical_analysis + financial_analysis + + + +``` + +3. Evaluator-Optimizer Pattern (Iterative Refinement): +```xml + + generate_content + + + content + Generated content + + RESULT + + + + + + + evaluate_content + + generate_content + + Evaluate the quality of generated content + + + approved + Content meets quality standards + If quality score >= threshold + + RESULT + + + + needs_improvement + Content needs improvement + If quality score < threshold + + GOTO + generate_content + + + + +``` + +IMPORTANT NOTES ON PATTERNS: +0. The above patterns are incomplete which some mandatory elements are missing due to the limitation of context length. In real-world, you could refer to the logic of the patterns to create a complete and correct workflow. + +1. If-Else Pattern: + - Use mutually exclusive conditions + - You can NOT place MORE THAN ONE OUTPUT with RESULT type + - Outputs determine which branch executes + +2. Parallelization Pattern: + - Multiple events can listen to the same parent event + - Aggregator event must list ALL parallel events in its listen section + - All parallel events must complete before aggregator executes + - Model of agents in every parallel event could be different + +3. Evaluator-Optimizer Pattern: + - Use GOTO action for iteration + - Include clear evaluation criteria in conditions + - Have both success and retry paths + - Consider adding maximum iteration limit in global_variables +""" + \ +r""" +EXAMPLE: + +User: I want to build a workflow that can help me to write a wikipiead-like article about the user's topic. It should: +1. Search the web for the user's topic. +2. Write an outline for the user's topic. +3. Evaluate the outline. If the outline is not good enough, repeat the outline step, otherwise, continue to write the article. +4. Write the article. + +The form should be: + + wiki_article_workflow + + user_topic + The user's topic that user wants to write a wikipiead-like article about. + + + article + The article that satisfies the user's request. + + + + Web Surfer Agent + This agent is used to search the web for the user's topic. + + + Outline Agent + This agent is used to write an outline for the user's topic. + + + Evaluator Agent + This agent is used to evaluate the outline of the user's topic. + + + Article Writer Agent + This agent is used to write the article for the user's topic. + + + + + + on_start + + + user_topic + The user's topic that user wants to write a wikipiead-like article about. + + + + + user_topic + The user's topic that user wants to write a wikipiead-like article about. + + RESULT + + + + + + on_search + + + user_topic + The user's topic that user wants to write a wikipiead-like article about. + + + + search the information about the topic and return the result. + + + + search_result + The search result of the user's topic. + + RESULT + + + + + on_start + + + Web Surfer Agent + claude-3-5-sonnet-20241022 + + + + on_outline + + + search_result + The search result of the user's topic. + + + + write an outline for the user's topic. + + + + outline + The outline of the user's topic. + + RESULT + + + + + on_start + + + Outline Agent + claude-3-5-sonnet-20241022 + + + + on_evaluate + + + outline + The outline of the user's topic. + + + + evaluate the outline of the user's topic. + + + + positive_feedback + The positive feedback of the outline of the user's topic. + + If the outline is good enough, give positive feedback. + + + RESULT + + + + negative_feedback + The negative feedback of the outline of the user's topic. + + If the outline is not good enough, give negative feedback. + + + GOTO + on_outline + + + + + on_outline + + + Evaluator Agent + claude-3-5-sonnet-20241022 + + + + on_write + + + outline + The outline of user's topic. + + + + write the article for the user's topic. + + + + article + The article of the user's topic. + + RESULT + + + + + on_evaluate + + + Article Writer Agent + claude-3-5-sonnet-20241022 + + + + + +GUIDELINES: +1. Each event should have clear inputs and outputs +2. Use conditions to handle different outcomes +3. Properly chain events using the listen element +4. Review steps should be included for quality control +5. Action types should be either RESULT or ABORT + +Follow these examples and guidelines to create appropriate workflow forms based on user requirements. +""" + return Agent( + name = "Workflow Former Agent", + model = model, + instructions = instructions, + ) + +if __name__ == "__main__": + from autoagent import MetaChain + agent = get_workflow_former_agent("claude-3-5-sonnet-20241022") + client = MetaChain() +# task_yaml = """\ +# I want to create a workflow that can help me to solving the math problem. + +# The workflow should: +# 2. Parallelize solving the math problem with the same `Math Solver Agent` using different language models (`gpt-4o-2024-08-06`, `claude-3-5-sonnet-20241022`, `deepseek/deepseek-chat`) +# 3. Aggregate the results from the `Math Solver Agent` and return the final result using majority voting. + +# Please create the form of this workflow in the XML format. +# """ + task_yaml = """\ +I want to create a workflow that can help me to solving the math problem. + +The workflow should: +1. The `Objective Extraction Agent` will extract the objective of the math problem. +2. The `Condition Extraction Agent` will extract the conditions of the math problem. +3. The `Math Solver Agent` will evaluate whether the conditions are enough to solve the math problem: if yes, solve the math problem; if no, return to the `Condition Extraction Agent` to extract more conditions. + +Please create the form of this workflow in the XML format. +""" + task_yaml = task_yaml + """\ +Directly output the form in the XML format. +""" + messages = [{"role": "user", "content": task_yaml}] + response = client.run(agent, messages) + print(response.messages[-1]["content"]) \ No newline at end of file diff --git a/autoagent/agents/meta_agent/worklow_form_complie.py b/autoagent/agents/meta_agent/worklow_form_complie.py new file mode 100644 index 0000000..b94584b --- /dev/null +++ b/autoagent/agents/meta_agent/worklow_form_complie.py @@ -0,0 +1,324 @@ +from pydantic import BaseModel, Field, field_validator, ValidationInfo, model_validator +from typing import List, Dict, Optional, Literal, Union +import xml.etree.ElementTree as ET +import re +# 基础模型 + +class WorkflowFormParseError(Exception): + """Exception raised when WorkflowForm failed to parse. + """ + + def __init__(self, message): + super().__init__(message) + +class WorkflowConstraintError(Exception): + """Exception raised when WorkflowForm failed to parse. Use this Exception to raise when the workflow form does not meet some specific constraints. + """ + + def __init__(self, message): + super().__init__(message) + +class KeyDescription(BaseModel): + key: str + description: str + +class Tool(BaseModel): + name: str + description: str + +class Action(BaseModel): + type: Literal["RESULT", "ABORT", "GOTO"] + value: Optional[str] = None + + @field_validator('value') + def validate_goto_value(cls, v, info: ValidationInfo): + if info.data.get('type') == 'GOTO' and not v: + raise WorkflowConstraintError("GOTO action must have a value") + return v + +class Output(BaseModel): + key: str + description: str + condition: Optional[str] = None + action: Action + + @field_validator('condition') + def validate_condition(cls, v, info: ValidationInfo): + """验证condition的存在性""" + outputs_info = info.data.get('_outputs_info', {}) + if outputs_info.get('multiple_outputs', False) and not v: + raise WorkflowConstraintError("Multiple outputs must each have a condition") + return v + +class Event(BaseModel): + name: str + inputs: Optional[List[KeyDescription]] = None # 修改这里 + task: Optional[str] = None # 修改为可选 + outputs: List[Output] + listen: Optional[List[str]] = None + agent: Optional[Dict[str, str]] = None # 修改为可选 + + @field_validator('task') + def validate_task(cls, v, info: ValidationInfo): + """验证非on_start事件必须有task""" + if info.data.get('name') != 'on_start' and not v: + raise WorkflowConstraintError("Non-start events must have a task") + return v + @field_validator('agent') + def validate_agent(cls, v, info: ValidationInfo): + """验证非on_start事件必须有agent""" + if info.data.get('name') != 'on_start' and not v: + raise WorkflowConstraintError("Non-start events must have an agent") + return v + + @field_validator('listen') + def validate_listen(cls, v, info: ValidationInfo): + """验证on_start事件不能有listen""" + if info.data.get('name') == 'on_start' and v: + raise WorkflowConstraintError("Start event cannot have listen elements") + return v + + @field_validator('name') + def validate_start_event(cls, v, info: ValidationInfo): + """验证起始事件的名称""" + if info.data.get('is_start_event', False) and v != "on_start": + raise WorkflowConstraintError("Start event must be named 'on_start'") + return v + @field_validator('outputs') + def validate_start_event_outputs(cls, v, info: ValidationInfo): + """验证on_start事件的输出必须与输入相同""" + if info.data.get('name') == 'on_start': + inputs = info.data.get('inputs', []) + if len(v) != len(inputs): + raise WorkflowConstraintError("Start event outputs must match inputs") + for output, input in zip(v, inputs): + if output.key != input.key or output.description != input.description: + raise WorkflowConstraintError("Start event output must match input") + return v + + @field_validator('outputs') + def validate_outputs(cls, v): + """验证输出的合法性""" + result_outputs = [out for out in v if out.action.type == "RESULT"] + if len(result_outputs) > 1: + raise WorkflowConstraintError("Cannot have more than one RESULT type output") + return v + + @model_validator(mode='after') + def validate_event_constraints(self) -> 'Event': + """验证事件的所有约束""" + # 如果是 on_start event，跳过输入验证 + if self.name == "on_start": + return self + + # 验证非on_start事件的输入 + if self.inputs is None: + raise WorkflowConstraintError(f"Event '{self.name}': Non-start events must have inputs") + + # 验证listen是否存在 + if self.listen is None: + raise WorkflowConstraintError(f"Event '{self.name}': Non-start events must have listen events") + + # 验证输入数量 + if len(self.inputs) != len(self.listen): + raise WorkflowConstraintError( + f"Event '{self.name}': Number of inputs ({len(self.inputs)}) must match number of listen events ({len(self.listen)})" + ) + + return self + +class Agent(BaseModel): + name: str + description: str + category: Literal["existing", "new"] + tools: Optional[List[Tool]] = None + + @field_validator('tools') + def validate_tools(cls, v, info: ValidationInfo): + """验证tools的存在性""" + if info.data.get('category') == 'existing' and v: + raise WorkflowConstraintError("Existing agents should not have tools defined") + return v + +class WorkflowForm(BaseModel): + name: str + system_input: KeyDescription + system_output: KeyDescription + global_variables: Dict[str, str] = Field(default_factory=dict) + agents: List[Agent] + events: List[Event] + + @field_validator('events') + def validate_events(cls, v): + """验证事件流的合法性""" + # 验证是否有且仅有一个on_start事件 + start_events = [e for e in v if e.name == "on_start"] + if len(start_events) != 1: + raise WorkflowConstraintError("Must have exactly one 'on_start' event") + + # 验证事件监听的合法性 + event_names = {e.name for e in v} + for event in v: + if event.listen: + for listened_event in event.listen: + if listened_event not in event_names: + raise WorkflowConstraintError(f"Event {event.name} listens to non-existent event {listened_event}") + return v + @model_validator(mode='after') + def validate_event_order(self) -> 'WorkflowForm': + """验证事件的监听顺序： + 1. 事件只能监听在它之前定义的事件 + 2. 不能有循环依赖 + """ + # 创建事件名称到索引的映射 + event_indices = {event.name: idx for idx, event in enumerate(self.events)} + + # 验证每个事件的监听关系 + for idx, event in enumerate(self.events): + if event.listen: + for listened_event_name in event.listen: + # 检查被监听的事件是否存在 + if listened_event_name not in event_indices: + raise WorkflowConstraintError( + f"Event '{event.name}': Referenced listen event '{listened_event_name}' not found" + ) + + # 检查是否监听了后面的事件 + listened_idx = event_indices[listened_event_name] + if listened_idx >= idx: + raise WorkflowConstraintError( + f"Event '{event.name}' cannot listen to event '{listened_event_name}' " + f"because it appears later in the workflow or creates a cycle" + ) + + return self + +class XMLParser: + @staticmethod + def parse_key_description(elem: ET.Element) -> KeyDescription: + return KeyDescription( + key=elem.find('key').text.strip(), + description=elem.find('description').text.strip() + ) + + @staticmethod + def parse_action(elem: ET.Element) -> Action: + action_elem = elem.find('action') + return Action( + type=action_elem.find('type').text.strip(), + value=action_elem.find('value').text.strip() if action_elem.find('value') is not None else None + ) + + @staticmethod + def parse_output(elem: ET.Element, multiple_outputs: bool) -> Output: + return Output( + key=elem.find('key').text.strip(), + description=elem.find('description').text.strip(), + condition=elem.find('condition').text.strip() if elem.find('condition') is not None else None, + action=XMLParser.parse_action(elem), + _outputs_info={'multiple_outputs': multiple_outputs} + ) + + @staticmethod + def parse_event(elem: ET.Element, is_start: bool = False) -> Event: + name = elem.find('name').text.strip() + is_start = name == 'on_start' + + outputs_elem = elem.find('outputs') + multiple_outputs = len(outputs_elem.findall('output')) > 1 + + listen_elem = elem.find('listen') + listen = [e.text.strip() for e in listen_elem.findall('event')] if listen_elem is not None and not is_start else None + + agent_elem = elem.find('agent') + agent = { + "name": agent_elem.find('name').text.strip(), + "model": agent_elem.find('model').text.strip() + } if agent_elem is not None and not is_start else None + + inputs_elem = elem.find('inputs') + inputs = [XMLParser.parse_key_description(input_elem) + for input_elem in inputs_elem.findall('input')] if inputs_elem is not None else None + task_elem = elem.find('task') + task = task_elem.text.strip() if task_elem is not None and not is_start else None + + return Event( + name=name, + inputs=inputs, + task=task, + outputs=[XMLParser.parse_output(out, multiple_outputs) + for out in outputs_elem.findall('output')], + listen=listen, + agent=agent, + is_start_event=is_start + ) + + @staticmethod + def parse_agent(elem: ET.Element) -> Agent: + tools_elem = elem.find('tools') + tools = None + if tools_elem is not None: + tools = [Tool( + name=tool.find('name').text.strip(), + description=tool.find('description').text.strip() + ) for tool in tools_elem.findall('tool')] + + return Agent( + name=elem.find('name').text.strip(), + description=elem.find('description').text.strip(), + category=elem.get('category'), + tools=tools + ) + + @classmethod + def parse_xml(cls, xml_content: str) -> WorkflowForm: + root = ET.fromstring(xml_content) + workflow_name = root.get('name') + if not workflow_name: + # If name attribute doesn't exist, try to find name element + name_elem = root.find('name') + workflow_name = name_elem.text.strip() if name_elem is not None else "Unnamed Workflow" + + return WorkflowForm( + name=workflow_name, + system_input=cls.parse_key_description(root.find('system_input')), + system_output=cls.parse_key_description(root.find('system_output')), + global_variables={var.find('key').text.strip(): var.find('value').text.strip() + for var in root.find('global_variables').findall('variable')} + if root.find('global_variables') is not None else {}, + agents=[cls.parse_agent(agent) for agent in root.findall('.//agents/agent')], + events=[cls.parse_event(event, event.find('name').text.strip() == 'on_start') + for event in root.findall('.//events/event')] + ) + +def extract_workflow_content(text): + pattern = r'(.*?)' + # re.DOTALL 让 . 也能匹配换行符 + match = re.search(pattern, text, re.DOTALL) + if match: + return match.group(1) + else: + raise WorkflowFormParseError("The workflow XML form is not correct. The workflow XML form should be wrapped by ... tags.") + +def parse_workflow_form(xml_content: str) -> Optional[WorkflowForm]: + """ + 读取并解析workflow form XML文件 + + Args: + xml_content: XML文件内容 + + Returns: + 解析后的WorkflowForm对象，如果解析失败返回None + """ + try: + workflow_content = extract_workflow_content(xml_content) + return XMLParser.parse_xml(workflow_content) + except WorkflowFormParseError as e: + return f"The Error to extract workflow content: {e}" + except WorkflowConstraintError as e: + return f"The generated workflow form MUST meet all the constraints in the given instructions, but the constraints are not met: {e}" + except ET.ParseError as e: + return f"The Error parsing XML workflow form: {e}" + except Exception as e: + return f"Unexpected error: {e}" + diff --git a/autoagent/agents/system_agent/filesurfer_agent.py b/autoagent/agents/system_agent/filesurfer_agent.py new file mode 100644 index 0000000..8653bea --- /dev/null +++ b/autoagent/agents/system_agent/filesurfer_agent.py @@ -0,0 +1,43 @@ +from autoagent.types import Agent +from autoagent.registry import register_agent +from autoagent.tools import open_local_file, page_up_markdown, page_down_markdown, find_on_page_ctrl_f, find_next, visual_question_answering +from autoagent.tools.file_surfer_tool import with_env +from autoagent.environment.markdown_browser import RequestsMarkdownBrowser +import time +from inspect import signature +from constant import LOCAL_ROOT, DOCKER_WORKPLACE_NAME +@register_agent(name = "File Surfer Agent", func_name="get_filesurfer_agent") +def get_filesurfer_agent(model: str = "gpt-4o", **kwargs): + + def handle_mm_func(tool_name, tool_args): + return f"After using tool `{tool_name}({tool_args})`, I have opened the image I want to see and prepared a question according to the image. Please answer the question based on the image." + def instructions(context_variables): + file_env: RequestsMarkdownBrowser = context_variables.get("file_env", None) + assert file_env is not None, "file_env is required" + return \ +f""" +You are a file surfer agent that can handle local files. + +You can only access the files in the folder `{file_env.docker_workplace}` and when you want to open a file, you should use absolute path from root like `{file_env.docker_workplace}/...`. + +Note that `open_local_file` can read a file as markdown text and ask questions about it. And `open_local_file` can handle the following file extensions: [".html", ".htm", ".xlsx", ".pptx", ".wav", ".mp3", ".flac", ".pdf", ".docx"], and all other types of text files. + +But IT DOES NOT HANDLE IMAGES, you should use `visual_question_answering` to see the image. + +If the converted markdown text has more than 1 page, you can use `page_up`, `page_down`, `find_on_page_ctrl_f`, `find_next` to navigate through the pages. + +When you think you have completed the task the `System Triage Agent` asked you to do, you should use `transfer_back_to_triage_agent` to transfer the conversation back to the `System Triage Agent`. And you should not stop to try to solve the user's request by transferring to `System Triage Agent` only until the task is completed. + +If you are unable to open the file, you can transfer the conversation back to the `System Triage Agent`, and let the `Coding Agent` try to solve the problem by coding. +""" + tool_list = [open_local_file, page_up_markdown, page_down_markdown, find_on_page_ctrl_f, find_next, visual_question_answering] + return Agent( + name="File Surfer Agent", + model=model, + instructions=instructions, + functions=tool_list, + handle_mm_func=handle_mm_func, + tool_choice = "required", + parallel_tool_calls = False + ) + diff --git a/autoagent/agents/system_agent/programming_agent.py b/autoagent/agents/system_agent/programming_agent.py new file mode 100644 index 0000000..5ef6e1d --- /dev/null +++ b/autoagent/agents/system_agent/programming_agent.py @@ -0,0 +1,98 @@ +from autoagent.types import Agent +from autoagent.tools import ( + gen_code_tree_structure, execute_command, read_file, create_file, write_file, list_files, create_directory, run_python, terminal_page_up, terminal_page_down, terminal_page_to +) +from autoagent.util import make_message, make_tool_message +from autoagent.registry import register_agent, register_plugin_agent +from constant import LOCAL_ROOT, DOCKER_WORKPLACE_NAME +from autoagent.environment import DockerEnv, BrowserEnv, LocalEnv +from typing import Union +from inspect import signature +def examples(context_variables): + working_dir = context_variables.get("working_dir", None) + examples_list = [] + examples_list.extend(make_message('user', "Create a list of numbers from 1 to 10, and display them in a web page at port 5000.")) + examples_list.extend(make_message('assistant', "I should first use create_file to write the python code into a file named 'app.py' for starting a web server")) + examples_list.extend(make_tool_message(create_file, {'path': f"/{working_dir}/app.py", + 'content': """ +from flask import Flask +app = Flask(__name__) + +@app.route('/') +def index(): + numbers = list(range(1, 11)) + return str(numbers) + +if __name__ == '__main__': + app.run(port=5000)"""}, f"File created at: /{working_dir}/app.py")) + examples_list.extend(make_message('assistant', 'I have created a Python file `app.py` that will display a list of numbers from 1 to 10 when you run it. Let me run the Python file for you using `run_python`')) + examples_list.extend(make_tool_message(run_python, {'code_path': f"/{working_dir}/app.py"}, f""" +Traceback (most recent call last): + File "/{working_dir}/app.py", line 2, in + from flask import Flask +ModuleNotFoundError: No module named 'flask'""")) + examples_list.extend(make_message('assistant', "It seems that Flask is not installed. Let me install Flask for you using `execute_command` by the command: pip install flask")) + examples_list.extend(make_tool_message(execute_command, {'command': 'pip install flask'}, """Defaulting to user installation because normal site-packages is not writeable +Collecting flask + Using cached flask-3.0.3-py3-none-any.whl (101 kB) +Collecting blinker>=1.6.2 + Using cached blinker-1.7.0-py3-none-any.whl (13 kB) +Collecting Werkzeug>=3.0.0 + Using cached werkzeug-3.0.2-py3-none-any.whl (226 kB) +Collecting click>=8.1.3 + Using cached click-8.1.7-py3-none-any.whl (97 kB) +Collecting itsdangerous>=2.1.2 + Using cached itsdangerous-2.2.0-py3-none-any.whl (16 kB) +Requirement already satisfied: Jinja2>=3.1.2 in /home/openhands/.local/lib/python3.10/site-packages (from flask) (3.1.3) +Requirement already satisfied: MarkupSafe>=2.0 in /home/openhands/.local/lib/python3.10/site-packages (from Jinja2>=3.1.2->flask) (2.1.5) +Installing collected packages: Werkzeug, itsdangerous, click, blinker, flask +Successfully installed Werkzeug-3.0.2 blinker-1.7.0 click-8.1.7 flask-3.0.3 itsdangerous-2.2.0""")) + examples_list.extend(make_message('assistant', "Now that Flask is installed, let me run the Python file again using `run_python`")) + examples_list.extend(make_tool_message(run_python, {'code_path': f"/{working_dir}/app.py"}, """* Serving Flask app 'app' + * Debug mode: off +WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead. + * Running on http://127.0.0.1:5000 +Press CTRL+C to quit""")) + examples_list.extend(make_message('assistant', """The server is running on port 5000 with PID 124. You can access the list of numbers by visiting http://127.0.0.1:5000. Your task is completed.""")) + return examples_list + +@register_agent(name= "Coding Agent", func_name="get_coding_agent") +@register_plugin_agent(name= "Coding Agent", func_name="get_coding_agent") +def get_coding_agent(model: str, **kwargs): + def instructions(context_variables): + code_env: Union[DockerEnv, LocalEnv] = context_variables.get("code_env", LocalEnv()) + return f"""You are a helpful programming assistant that can write and execute code. You are working in the folder: `{code_env.docker_workplace}`, and you can only access the files in this folder. + + Your can leverage your capabilities by using the specific functions listed below: + + 1. Creating project structures based on the user requirement using function `create_directory`. + 2. Writing clean, efficient, and well-documented code using function `create_file` and `write_file`. + 3. You must run python scripts using function `run_python` rather than using the `execute_command` function. + 4. Exam the project to re-use the existing code snippets as much as possible, you may need to use + functions like `list_files`, `read_file` and `write_file`. + 5. Writing the code into the file when creating new files, do not create empty files. + 6. Before you write code into the existing files, you should first read the file content using function `read_file` and reserve the original content as much as possible. + 7. Decide whether the task requires execution and debugging before moving to the next or not. + 8. Generate the commands to run and test the current task, and the dependencies list for this task. + 9. You only write Python scripts, don't write Jupiter notebooks which require interactive execution. + 10. Note that every path you read, write, or search should be the absolute path (starting with '/'). + 11. If you should use programming other than Python, you should use the `write_file` function to write the code into a file, and then use the `execute_command` function to run the code. + 12. If the terminal output is too long, you should use `terminal_page_up` to move the viewport up, `terminal_page_down` to move the viewport down, `terminal_page_to` to move the viewport to the specific page of terminal where the meaningful content is. + + Note that you can use this agent to make complex computation, write a api request, and anything else that can be done by writing code. + + When you think you have completed the task the `System Triage Agent` asked you to do, you should use `transfer_back_to_triage_agent` to transfer the conversation back to the `System Triage Agent`. And you should not stop to try to solve the user's request by transferring to `System Triage Agent` only until the task is completed. + + [IMPORTANT] You can only complete the task by coding. Talk is cheap, show me the code with tools. + """ + tool_list = [gen_code_tree_structure, execute_command, read_file, create_file, write_file, list_files, create_directory, run_python, terminal_page_up, terminal_page_down, terminal_page_to] + + return Agent( + name="Coding Agent", + model=model, + instructions=instructions, + functions=tool_list, + # examples=examples, + tool_choice = "required", + parallel_tool_calls = False + ) \ No newline at end of file diff --git a/autoagent/agents/system_agent/system_triage_agent.py b/autoagent/agents/system_agent/system_triage_agent.py new file mode 100644 index 0000000..3fa5706 --- /dev/null +++ b/autoagent/agents/system_agent/system_triage_agent.py @@ -0,0 +1,66 @@ +from .filesurfer_agent import get_filesurfer_agent +from .programming_agent import get_coding_agent +from .websurfer_agent import get_websurfer_agent +from autoagent.registry import register_agent +from autoagent.types import Agent, Result +from autoagent.tools.inner import case_resolved, case_not_resolved + +@register_agent(name = "System Triage Agent", func_name="get_system_triage_agent") +def get_system_triage_agent(model: str, **kwargs): + """ + This is the `System Triage Agent`, it can help the user to determine which agent is best suited to handle the user's request under the current context, and transfer the conversation to that agent. + + Args: + model: The model to use for the agent. + **kwargs: Additional keyword arguments, `file_env`, `web_env` and `code_env` are required. + """ + filesurfer_agent = get_filesurfer_agent(model) + websurfer_agent = get_websurfer_agent(model) + coding_agent = get_coding_agent(model) + instructions = \ +f"""You are a helpful assistant that can help the user with their request. +Based on the state of solving user's task, your responsibility is to determine which agent is best suited to handle the user's request under the current context, and transfer the conversation to that agent. And you should not stop to try to solve the user's request by transferring to another agent only until the task is completed. + +There are three agents you can transfer to: +1. use `transfer_to_filesurfer_agent` to transfer to {filesurfer_agent.name}, it can help you to open any type of local files and browse the content of them. +2. use `transfer_to_websurfer_agent` to transfer to {websurfer_agent.name}, it can help you to open any website and browse any content on it. +3. use `transfer_to_coding_agent` to transfer to {coding_agent.name}, it can help you to write code to solve the user's request, especially some complex tasks. +""" + tool_choice = "required" + tools = [case_resolved, case_not_resolved] if tool_choice == "required" else [] + system_triage_agent = Agent( + name="System Triage Agent", + model=model, + instructions=instructions, + functions=tools, + tool_choice = tool_choice, + parallel_tool_calls = False, + ) + def transfer_to_filesurfer_agent(sub_task_description: str): + """ + Args: + sub_task_description: The detailed description of the sub-task that the `System Triage Agent` will ask the `File Surfer Agent` to do. + """ + return Result(value=sub_task_description, agent=filesurfer_agent) + def transfer_to_websurfer_agent(sub_task_description: str): + return Result(value=sub_task_description, agent=websurfer_agent) + def transfer_to_coding_agent(sub_task_description: str): + return Result(value=sub_task_description, agent=coding_agent) + def transfer_back_to_triage_agent(task_status: str): + """ + Args: + task_status: The detailed description of the task status after a sub-agent has finished its sub-task. A sub-agent can use this tool to transfer the conversation back to the `System Triage Agent` only when it has finished its sub-task. + """ + return Result(value=task_status, agent=system_triage_agent) + system_triage_agent.agent_teams = { + filesurfer_agent.name: transfer_to_filesurfer_agent, + websurfer_agent.name: transfer_to_websurfer_agent, + coding_agent.name: transfer_to_coding_agent + } + system_triage_agent.functions.extend([transfer_to_filesurfer_agent, transfer_to_websurfer_agent, transfer_to_coding_agent]) + filesurfer_agent.functions.append(transfer_back_to_triage_agent) + websurfer_agent.functions.append(transfer_back_to_triage_agent) + coding_agent.functions.append(transfer_back_to_triage_agent) + return system_triage_agent + + \ No newline at end of file diff --git a/autoagent/agents/system_agent/websurfer_agent.py b/autoagent/agents/system_agent/websurfer_agent.py new file mode 100644 index 0000000..d63e51d --- /dev/null +++ b/autoagent/agents/system_agent/websurfer_agent.py @@ -0,0 +1,37 @@ +from autoagent.types import Agent +from autoagent.registry import register_agent +from autoagent.tools import click, page_down, page_up, history_back, history_forward, web_search, input_text, sleep, visit_url, get_page_markdown +from autoagent.tools.web_tools import with_env +from autoagent.environment.browser_env import BrowserEnv +import time +from constant import DOCKER_WORKPLACE_NAME, LOCAL_ROOT +@register_agent(name = "Web Surfer Agent", func_name="get_websurfer_agent") +def get_websurfer_agent(model: str = "gpt-4o", **kwargs): + + def handle_mm_func(tool_name, tool_args): + return f"After take last action `{tool_name}({tool_args})`, the image of current page is shown below. Please take next action based on the image, the current state of the page as well as previous actions and observations." + def instructions(context_variables): + web_env: BrowserEnv = context_variables.get("web_env", None) + assert web_env is not None, "web_env is required" + return \ +f"""Review the current state of the page and all other information to find the best possible next action to accomplish your goal. Your answer will be interpreted and executed by a program, make sure to follow the formatting instructions. + +Note that if you want to analyze the YouTube video, Wikipedia page, or other pages that contain media content, or you just want to analyze the text content of the page in a more detailed way, you should use `get_page_markdown` tool to convert the page information to markdown text. And when browsing the web, if you have downloaded any files, the path of the downloaded files will be `{web_env.docker_workplace}/downloads`, and you CANNOT open the downloaded files directly, you should transfer back to the `System Triage Agent`, and let `System Triage Agent` to transfer to `File Surfer Agent` to open the downloaded files. + +When you think you have completed the task the `System Triage Agent` asked you to do, you should use `transfer_back_to_triage_agent` to transfer the conversation back to the `System Triage Agent`. And you should not stop to try to solve the user's request by transferring to `System Triage Agent` only until the task is completed. +""" + + tool_list = [click, page_down, page_up, history_back, history_forward, web_search, input_text, sleep, visit_url, get_page_markdown] + return Agent( + name="Web Surfer Agent", + model=model, + instructions=instructions, + functions=tool_list, + handle_mm_func=handle_mm_func, + tool_choice = "required", + parallel_tool_calls = False + ) + +""" +Note that when you need to download something, you should first know the url of the file, and then use the `visit_url` tool to download the file. For example, if you want to download paper from 'https://arxiv.org/abs/2310.13023', you should use `visit_url('url'='https://arxiv.org/pdf/2310.13023.pdf')`. +""" \ No newline at end of file diff --git a/autoagent/agents/tool_retriver_agent.py b/autoagent/agents/tool_retriver_agent.py new file mode 100644 index 0000000..fff5524 --- /dev/null +++ b/autoagent/agents/tool_retriver_agent.py @@ -0,0 +1,21 @@ +from autoagent.types import Agent +from autoagent.tools import ( + get_api_plugin_tools_doc +) +from autoagent.util import make_message, make_tool_message +from autoagent.registry import register_agent +@register_agent(name = "Tool Retriver Agent", func_name="get_tool_retriver_agent") +def get_tool_retriver_agent(model: str): + def instructions(context_variables): + return \ +f""" +You are a tool retriver agent. +You are given a task instruction, and you need to retrieve the tool docs for the task using function `get_tool_doc`. +Note that if you want to complete the task, you may need to use more than one tool, so you should retrieve the tool docs for all the tools you may need. Finally, you should give a merged tool doc consisting of all the tool docs you retrieved, and the implementation code of each tool should be included in the tool doc. +""" + return Agent( + name="Tool Retriver Agent", + model=model, + instructions=instructions, + functions=[get_api_plugin_tools_doc], + ) \ No newline at end of file diff --git a/autoagent/cli.py b/autoagent/cli.py new file mode 100644 index 0000000..0a0862e --- /dev/null +++ b/autoagent/cli.py @@ -0,0 +1,426 @@ +import click +import importlib +from autoagent import MetaChain +from autoagent.util import debug_print +import asyncio +from constant import DOCKER_WORKPLACE_NAME +from autoagent.io_utils import read_yaml_file, get_md5_hash_bytext, read_file +from autoagent.environment.utils import setup_metachain +from autoagent.types import Response +from autoagent import MetaChain +from autoagent.util import ask_text, single_select_menu, print_markdown, debug_print, UserCompleter +from prompt_toolkit import PromptSession +from prompt_toolkit.completion import Completer, Completion +from prompt_toolkit.formatted_text import HTML +from prompt_toolkit.styles import Style +from rich.progress import Progress, SpinnerColumn, TextColumn +import json +import argparse +from datetime import datetime +from autoagent.agents.meta_agent import tool_editor, agent_editor +from autoagent.tools.meta.edit_tools import list_tools +from autoagent.tools.meta.edit_agents import list_agents +from loop_utils.font_page import MC_LOGO, version_table, NOTES, GOODBYE_LOGO +from rich.live import Live +from autoagent.environment.docker_env import DockerEnv, DockerConfig, check_container_ports +from autoagent.environment.local_env import LocalEnv +from autoagent.environment.browser_env import BrowserEnv +from autoagent.environment.markdown_browser import RequestsMarkdownBrowser +from evaluation.utils import update_progress, check_port_available, run_evaluation, clean_msg +import os +import os.path as osp +from autoagent.agents import get_system_triage_agent +from autoagent.logger import LoggerManager, MetaChainLogger +from rich.console import Console +from rich.markdown import Markdown +from rich.table import Table +from rich.columns import Columns +from rich.text import Text +from rich.panel import Panel +import re +from autoagent.cli_utils.metachain_meta_agent import meta_agent +from autoagent.cli_utils.metachain_meta_workflow import meta_workflow +from autoagent.cli_utils.file_select import select_and_copy_files +from evaluation.utils import update_progress, check_port_available, run_evaluation, clean_msg +from constant import COMPLETION_MODEL +@click.group() +def cli(): + """The command line interface for autoagent""" + pass + +@cli.command() +@click.option('--model', default='gpt-4o-2024-08-06', help='the name of the model') +@click.option('--agent_func', default='get_dummy_agent', help='the function to get the agent') +@click.option('--query', default='...', help='the user query to the agent') +@click.argument('context_variables', nargs=-1) +def agent(model: str, agent_func: str, query: str, context_variables): + """ + Run an agent with a given model, agent function, query, and context variables. + Args: + model (str): The name of the model. + agent_func (str): The function to get the agent. + query (str): The user query to the agent. + context_variables (list): The context variables to pass to the agent. + Usage: + mc agent --model=gpt-4o-2024-08-06 --agent_func=get_weather_agent --query="What is the weather in Tokyo?" city=Tokyo unit=C timestamp=2024-01-01 + """ + context_storage = {} + for arg in context_variables: + if '=' in arg: + key, value = arg.split('=', 1) + context_storage[key] = value + agent_module = importlib.import_module(f'autoagent.agents') + try: + agent_func = getattr(agent_module, agent_func) + except AttributeError: + raise ValueError(f'Agent function {agent_func} not found, you shoud check in the `autoagent.agents` directory for the correct function name') + agent = agent_func(model) + mc = MetaChain() + messages = [ + {"role": "user", "content": query} + ] + response = mc.run(agent, messages, context_storage, debug=True) + debug_print(True, response.messages[-1]['content'], title = f'Result of running {agent.name} agent', color = 'pink3') + return response.messages[-1]['content'] + +@cli.command() +@click.option('--workflow_name', default=None, help='the name of the workflow') +@click.option('--system_input', default='...', help='the user query to the agent') +def workflow(workflow_name: str, system_input: str): + """命令行函数的同步包装器""" + return asyncio.run(async_workflow(workflow_name, system_input)) + +async def async_workflow(workflow_name: str, system_input: str): + """异步实现的workflow函数""" + workflow_module = importlib.import_module(f'autoagent.workflows') + try: + workflow_func = getattr(workflow_module, workflow_name) + except AttributeError: + raise ValueError(f'Workflow function {workflow_name} not found...') + + result = await workflow_func(system_input) # 使用 await 等待异步函数完成 + debug_print(True, result, title=f'Result of running {workflow_name} workflow', color='pink3') + return result + +def clear_screen(): + console = Console() + console.print("[bold green]Coming soon...[/bold green]") + print('\033[u\033[J\033[?25h', end='') # Restore cursor and clear everything after it, show cursor +def get_config(container_name, port, test_pull_name="main", git_clone=False): + container_name = container_name + + port_info = check_container_ports(container_name) + if port_info: + port = port_info[0] + else: + # while not check_port_available(port): + # port += 1 + # 使用文件锁来确保端口分配的原子性 + import filelock + lock_file = os.path.join(os.getcwd(), ".port_lock") + lock = filelock.FileLock(lock_file) + + with lock: + port = port + while not check_port_available(port): + port += 1 + print(f'{port} is not available, trying {port+1}') + # 立即标记该端口为已使用 + with open(os.path.join(os.getcwd(), f".port_{port}"), 'w') as f: + f.write(container_name) + local_root = os.path.join(os.getcwd(), f"workspace_meta_showcase", f"showcase_{container_name}") + os.makedirs(local_root, exist_ok=True) + docker_config = DockerConfig( + workplace_name=DOCKER_WORKPLACE_NAME, + container_name=container_name, + communication_port=port, + conda_path='/root/miniconda3', + local_root=local_root, + test_pull_name=test_pull_name, + git_clone=git_clone + ) + return docker_config +def create_environment(docker_config: DockerConfig): + """ + 1. create the code environment + 2. create the web environment + 3. create the file environment + """ + code_env = DockerEnv(docker_config) + code_env.init_container() + + web_env = BrowserEnv(browsergym_eval_env = None, local_root=docker_config.local_root, workplace_name=docker_config.workplace_name) + file_env = RequestsMarkdownBrowser(viewport_size=1024 * 5, local_root=docker_config.local_root, workplace_name=docker_config.workplace_name, downloads_folder=os.path.join(docker_config.local_root, docker_config.workplace_name, "downloads")) + + return code_env, web_env, file_env + +def create_environment_local(docker_config: DockerConfig): + """ + 1. create the code environment + 2. create the web environment + 3. create the file environment + """ + code_env = LocalEnv(docker_config) + + web_env = BrowserEnv(browsergym_eval_env = None, local_root=docker_config.local_root, workplace_name=docker_config.workplace_name) + file_env = RequestsMarkdownBrowser(viewport_size=1024 * 5, local_root=docker_config.local_root, workplace_name=docker_config.workplace_name, downloads_folder=os.path.join(docker_config.local_root, docker_config.workplace_name, "downloads")) + + return code_env, web_env, file_env + +def update_guidance(context_variables): + console = Console() + + # print the logo + logo_text = Text(MC_LOGO, justify="center") + console.print(Panel(logo_text, style="bold salmon1", expand=True)) + console.print(version_table) + console.print(Panel(NOTES,title="Important Notes", expand=True)) + +@cli.command(name='main') # 修改这里，使用连字符 +@click.option('--container_name', default='auto_agent', help='the function to get the agent') +@click.option('--port', default=12347, help='the port to run the container') +@click.option('--test_pull_name', default='autoagent_mirror', help='the name of the test pull') +@click.option('--git_clone', default=True, help='whether to clone a mirror of the repository') +@click.option('--local_env', default=False, help='whether to use local environment') +def main(container_name: str, port: int, test_pull_name: str, git_clone: bool, local_env: bool): + """ + Run deep research with a given model, container name, port + """ + model = COMPLETION_MODEL + print('\033[s\033[?25l', end='') # Save cursor position and hide cursor + with Progress( + SpinnerColumn(), + TextColumn("[progress.description]{task.description}"), + transient=True # 这会让进度条完成后消失 + ) as progress: + task = progress.add_task("[cyan]Initializing...", total=None) + + progress.update(task, description="[cyan]Initializing config...[/cyan]\n") + docker_config = get_config(container_name, port, test_pull_name, git_clone) + + progress.update(task, description="[cyan]Setting up logger...[/cyan]\n") + log_path = osp.join("casestudy_results", 'logs', f'agent_{container_name}_{model}.log') + LoggerManager.set_logger(MetaChainLogger(log_path = None)) + + progress.update(task, description="[cyan]Creating environment...[/cyan]\n") + if local_env: + code_env, web_env, file_env = create_environment_local(docker_config) + else: + code_env, web_env, file_env = create_environment(docker_config) + + progress.update(task, description="[cyan]Setting up autoagent...[/cyan]\n") + + clear_screen() + + context_variables = {"working_dir": docker_config.workplace_name, "code_env": code_env, "web_env": web_env, "file_env": file_env} + + # select the mode + while True: + update_guidance(context_variables) + mode = single_select_menu(['user mode', 'agent editor', 'workflow editor', 'exit'], "Please select the mode:") + match mode: + case 'user mode': + clear_screen() + user_mode(model, context_variables, False) + case 'agent editor': + clear_screen() + meta_agent(model, context_variables, False) + case 'workflow editor': + clear_screen() + meta_workflow(model, context_variables, False) + case 'exit': + console = Console() + logo_text = Text(GOODBYE_LOGO, justify="center") + console.print(Panel(logo_text, style="bold salmon1", expand=True)) + break + + +def user_mode(model: str, context_variables: dict, debug: bool = True): + logger = LoggerManager.get_logger() + console = Console() + system_triage_agent = get_system_triage_agent(model) + assert system_triage_agent.agent_teams != {}, "System Triage Agent must have agent teams" + messages = [] + agent = system_triage_agent + agents = {system_triage_agent.name.replace(' ', '_'): system_triage_agent} + for agent_name in system_triage_agent.agent_teams.keys(): + agents[agent_name.replace(' ', '_')] = system_triage_agent.agent_teams[agent_name]("placeholder").agent + agents["Upload_files"] = "select" + style = Style.from_dict({ + 'bottom-toolbar': 'bg:#333333 #ffffff', + }) + + # 创建会话 + session = PromptSession( + completer=UserCompleter(agents.keys()), + complete_while_typing=True, + style=style + ) + client = MetaChain(log_path=logger) + upload_infos = [] + while True: + # query = ask_text("Tell me what you want to do:") + query = session.prompt( + 'Tell me what you want to do (type "exit" to quit): ', + bottom_toolbar=HTML('Prompt: Enter @ to mention Agents') + ) + if query.strip().lower() == 'exit': + # logger.info('User mode completed. See you next time! :waving_hand:', color='green', title='EXIT') + + logo_text = "User mode completed. See you next time! :waving_hand:" + console.print(Panel(logo_text, style="bold salmon1", expand=True)) + break + words = query.split() + console.print(f"[bold green]Your request: {query}[/bold green]", end=" ") + for word in words: + if word.startswith('@') and word[1:] in agents.keys(): + # print(f"[bold magenta]{word}[bold magenta]", end=' ') + agent = agents[word.replace('@', '')] + else: + # print(word, end=' ') + pass + print() + + if hasattr(agent, "name"): + agent_name = agent.name + console.print(f"[bold green][bold magenta]@{agent_name}[/bold magenta] will help you, be patient...[/bold green]") + if len(upload_infos) > 0: + query = "{}\n\nUser uploaded files:\n{}".format(query, "\n".join(upload_infos)) + messages.append({"role": "user", "content": query}) + response = client.run(agent, messages, context_variables, debug=debug) + messages.extend(response.messages) + model_answer_raw = response.messages[-1]['content'] + + # attempt to parse model_answer + if model_answer_raw.startswith('Case resolved'): + model_answer = re.findall(r'(.*?)', model_answer_raw, re.DOTALL) + if len(model_answer) == 0: + model_answer = model_answer_raw + else: + model_answer = model_answer[0] + else: + model_answer = model_answer_raw + console.print(f"[bold green][bold magenta]@{agent_name}[/bold magenta] has finished with the response:\n[/bold green] [bold blue]{model_answer}[/bold blue]") + agent = response.agent + elif agent == "select": + code_env: DockerEnv = context_variables["code_env"] + local_workplace = code_env.local_workplace + docker_workplace = code_env.docker_workplace + files_dir = os.path.join(local_workplace, "files") + docker_files_dir = os.path.join(docker_workplace, "files") + os.makedirs(files_dir, exist_ok=True) + upload_infos.extend(select_and_copy_files(files_dir, console, docker_files_dir)) + agent = agents["System_Triage_Agent"] + else: + console.print(f"[bold red]Unknown agent: {agent}[/bold red]") + +@cli.command(name='deep-research') # 修改这里，使用连字符 +@click.option('--container_name', default='deepresearch', help='the function to get the agent') +@click.option('--port', default=12346, help='the port to run the container') +@click.option('--local_env', default=False, help='whether to use local environment') +def deep_research(container_name: str, port: int, local_env: bool): + """ + Run deep research with a given model, container name, port + """ + model = COMPLETION_MODEL + print('\033[s\033[?25l', end='') # Save cursor position and hide cursor + with Progress( + SpinnerColumn(), + TextColumn("[progress.description]{task.description}"), + transient=True # 这会让进度条完成后消失 + ) as progress: + task = progress.add_task("[cyan]Initializing...", total=None) + + progress.update(task, description="[cyan]Initializing config...[/cyan]\n") + docker_config = get_config(container_name, port) + + progress.update(task, description="[cyan]Setting up logger...[/cyan]\n") + log_path = osp.join("casestudy_results", 'logs', f'agent_{container_name}_{model}.log') + LoggerManager.set_logger(MetaChainLogger(log_path = None)) + + progress.update(task, description="[cyan]Creating environment...[/cyan]\n") + if local_env: + code_env, web_env, file_env = create_environment_local(docker_config) + else: + code_env, web_env, file_env = create_environment(docker_config) + + progress.update(task, description="[cyan]Setting up autoagent...[/cyan]\n") + + clear_screen() + + context_variables = {"working_dir": docker_config.workplace_name, "code_env": code_env, "web_env": web_env, "file_env": file_env} + + update_guidance(context_variables) + + logger = LoggerManager.get_logger() + console = Console() + system_triage_agent = get_system_triage_agent(model) + assert system_triage_agent.agent_teams != {}, "System Triage Agent must have agent teams" + messages = [] + agent = system_triage_agent + agents = {system_triage_agent.name.replace(' ', '_'): system_triage_agent} + for agent_name in system_triage_agent.agent_teams.keys(): + agents[agent_name.replace(' ', '_')] = system_triage_agent.agent_teams[agent_name]("placeholder").agent + agents["Upload_files"] = "select" + style = Style.from_dict({ + 'bottom-toolbar': 'bg:#333333 #ffffff', + }) + + # 创建会话 + session = PromptSession( + completer=UserCompleter(agents.keys()), + complete_while_typing=True, + style=style + ) + client = MetaChain(log_path=logger) + while True: + # query = ask_text("Tell me what you want to do:") + query = session.prompt( + 'Tell me what you want to do (type "exit" to quit): ', + bottom_toolbar=HTML('Prompt: Enter @ to mention Agents') + ) + if query.strip().lower() == 'exit': + # logger.info('User mode completed. See you next time! :waving_hand:', color='green', title='EXIT') + + logo_text = "See you next time! :waving_hand:" + console.print(Panel(logo_text, style="bold salmon1", expand=True)) + break + words = query.split() + console.print(f"[bold green]Your request: {query}[/bold green]", end=" ") + for word in words: + if word.startswith('@') and word[1:] in agents.keys(): + # print(f"[bold magenta]{word}[bold magenta]", end=' ') + agent = agents[word.replace('@', '')] + else: + # print(word, end=' ') + pass + print() + + if hasattr(agent, "name"): + agent_name = agent.name + console.print(f"[bold green][bold magenta]@{agent_name}[/bold magenta] will help you, be patient...[/bold green]") + messages.append({"role": "user", "content": query}) + response = client.run(agent, messages, context_variables, debug=False) + messages.extend(response.messages) + model_answer_raw = response.messages[-1]['content'] + + # attempt to parse model_answer + if model_answer_raw.startswith('Case resolved'): + model_answer = re.findall(r'(.*?)', model_answer_raw, re.DOTALL) + if len(model_answer) == 0: + model_answer = model_answer_raw + else: + model_answer = model_answer[0] + else: + model_answer = model_answer_raw + console.print(f"[bold green][bold magenta]@{agent_name}[/bold magenta] has finished with the response:\n[/bold green] [bold blue]{model_answer}[/bold blue]") + agent = response.agent + elif agent == "select": + code_env: DockerEnv = context_variables["code_env"] + local_workplace = code_env.local_workplace + files_dir = os.path.join(local_workplace, "files") + os.makedirs(files_dir, exist_ok=True) + select_and_copy_files(files_dir, console) + else: + console.print(f"[bold red]Unknown agent: {agent}[/bold red]") + \ No newline at end of file diff --git a/autoagent/cli_utils/file_select.py b/autoagent/cli_utils/file_select.py new file mode 100644 index 0000000..d363994 --- /dev/null +++ b/autoagent/cli_utils/file_select.py @@ -0,0 +1,57 @@ +import tkinter as tk +from tkinter import filedialog +import shutil +import os +from rich.console import Console + +def select_and_copy_files(dest_dir, console: Console, docker_files_dir: str): + # 创建 tkinter 根窗口但隐藏它 + root = tk.Tk() + root.withdraw() + + # 打开文件选择对话框 + files = filedialog.askopenfilenames( + title='Select files to copy', + filetypes=[ + # ('Text files', '*.txt'), + ('All files', '*.*'), + ('PDF files', '*.pdf'), + ('Docx files', '*.docx'), + ('Txt files', '*.txt'), + ('Zip files', '*.zip'), + ('Text files', '*.txt'), + ] + ) + + if not files: + print("No files selected") + return + + # 选择目标文件夹 + # dest_dir = filedialog.askdirectory( + # title='Select destination folder' + # ) + + if not dest_dir: + print("No destination folder selected") + return + + # 复制文件 + upload_infos = [] + for file_path in files: + file_name = os.path.basename(file_path) + dest_path = os.path.join(dest_dir, file_name) + docker_dest_path = os.path.join(docker_files_dir, file_name) + try: + shutil.copy2(file_path, dest_path) + msg = f"Uploaded: {file_name} to {docker_dest_path}" + upload_infos.append(msg) + console.print(f"[bold green]{msg}[/bold green]") + except Exception as e: + console.print(f"[bold red]Error uploading {file_name}: {e}[/bold red]") + + console.print(f"[bold green]Successfully uploaded {len(files)} files[/bold green]") + return upload_infos +if __name__ == "__main__": + dest_dir = "/Users/tangjiabin/Documents/reasoning/metachain/workspace_meta_showcase/showcase_nl2agent_showcase/workplace" + select_and_copy_files(dest_dir) \ No newline at end of file diff --git a/autoagent/cli_utils/metachain_meta_agent.py b/autoagent/cli_utils/metachain_meta_agent.py new file mode 100644 index 0000000..577bd6c --- /dev/null +++ b/autoagent/cli_utils/metachain_meta_agent.py @@ -0,0 +1,256 @@ +from autoagent import MetaChain +from autoagent.util import UserCompleter +from prompt_toolkit import PromptSession +from prompt_toolkit.formatted_text import HTML +from prompt_toolkit.styles import Style +from autoagent.logger import LoggerManager, MetaChainLogger +from rich.console import Console +from rich.panel import Panel +from autoagent.agents.meta_agent.agent_former import get_agent_former_agent +from autoagent.agents.meta_agent.tool_editor import get_tool_editor_agent +from autoagent.agents.meta_agent.agent_creator import get_agent_creator_agent +import re +from autoagent.agents.meta_agent.form_complie import parse_agent_form + + +def extract_agents_content(text): + pattern = r'(.*?)' + # re.DOTALL 让 . 也能匹配换行符 + match = re.search(pattern, text, re.DOTALL) + if match: + return match.group(1) + return None + +def agent_profiling(agent_former, client, messages, context_variables, requirements, debug): + messages.append({"role": "user", "content": requirements+ """ +Directly output the form in the XML format without ANY other text. +"""}) + + response = client.run(agent_former, messages, context_variables, debug=debug) + output_xml_form = response.messages[-1]["content"] + messages.extend(response.messages) + agent_form = None + + MAX_RETRY = 3 + for i in range(MAX_RETRY): + try: + output_xml_form = extract_agents_content(output_xml_form) + assert output_xml_form is not None, "No the XML form should be found in the output with the tag ...." + agent_form = parse_agent_form(output_xml_form) + break + except Exception as e: + print(f"Error parsing XML to agent form: {e}. Retry {i+1}/{MAX_RETRY}") + messages.append({"role": "user", "content": f"Error parsing XML to agent form: {e}\nNote that there are some special restrictions for creating agent form, please try again."}) + response = client.run(agent_former, messages, context_variables, debug=debug) + output_xml_form = response.messages[-1]["content"] + messages.extend(response.messages) + return agent_form, output_xml_form, messages + +def tool_editing(tool_editor_agent, client, messages, context_variables, agent_form, output_xml_form, debug, suggestions = ""): + def case_resolved(task_response: str, context_variables: dict): + """ + Use this tools when ALL desired tools are created and tested successfully. You can NOT use this tool if tools are not created or tested successfully by running the tools. + + Args: + task_response: the response of creating the tool which contains the completion status of the tool. + """ + return f"Case resolved. ALL desired tools are created and tested successfully. Details: {task_response}" + def case_not_resolved(task_response: str, context_variables: dict): + """ + Use this tools when you encounter irresistible errors after trying your best with multiple attempts for creating the desired tool. You can NOT use this tool before you have tried your best. + + Args: + task_response: the reason why the tool is not created or tested successfully. + """ + return f"Case not resolved. Some desired tools are not created or tested successfully. Details: {task_response}" + tool_editor_agent.functions.extend([case_resolved, case_not_resolved]) + MAX_RETRY = 3 + + if suggestions != "": + suggestions = "[IMPORTANT] Here are some suggestions for creating the tools: " + suggestions + + agents = agent_form.agents + new_tools = [] + for agent in agents: + if len(agent.tools.new) > 0: + + for idx, tool in enumerate(agent.tools.new): + new_tools.append(f"{idx+1}. Tool name: {tool.name}, Tool description: {tool.description}") + if len(new_tools) == 0: + return "Case resolved. ALL desired tools are created and tested successfully.", messages + new_tools_str = "\n".join(new_tools) + messages.append({"role": "user", "content": f"""\ +Your task is to create a list of new tools for me, the tools are: +{new_tools_str} +{suggestions} + +Please create these new tools for me, note that you can NOT stop util you have created all the tools and tested them using `run_tool` successfully. + +If ALL tools are created and tested successfully, you can stop and use `case_resolved` tool. Otherwise, you should continue to create the tools. After you have tried your best, you can use `case_not_resolved` tool to give the reason why the tool is not created or tested successfully. + +[IMPORTANT] ALL tools MUST be tested successfully by running the tools using `run_tool` before you stop. +"""}) + response = client.run(tool_editor_agent, messages, context_variables, debug=debug) + content = response.messages[-1]["content"] + for i in range(MAX_RETRY): + if content.startswith("Case resolved"): + return content, messages + messages.append({"role": "user", "content": f"""\ +Your task is to create a list of new tools for me, the tools are: +{new_tools_str} + +Please create these new tools for me, note that you can NOT stop util you have created all the tools and tested them using `run_tool` successfully. +The last attempt failed with the following error: {content}, please try again to create the tools. +"""}) + response = client.run(tool_editor_agent, messages, context_variables, debug=debug) + content = response.messages[-1]["content"] + if i == MAX_RETRY: + return f"{content}\nSome desired tools are not created or tested successfully with {MAX_RETRY} attempts.", messages + +def agent_editing(agent_creator_agent, client, messages, context_variables, agent_form, output_xml_form, requirements, task, debug, suggestions = ""): + MAX_RETRY = 3 + if suggestions != "": + suggestions = "[IMPORTANT] Here are some suggestions for creating the agent(s): " + suggestions + def case_resolved(task_response: str, context_variables: dict): + """ + Use this tools when the desired agent(s) is created and tested successfully. You can NOT use this tool if the agent(s) is not created or tested successfully by running the agent(s). + """ + return f"Case resolved. The desired agent(s) is created and tested successfully. : {task_response}" + def case_not_resolved(task_response: str, context_variables: dict): + """ + Use this tools when you encounter irresistible errors after trying your best with multiple attempts for creating the desired agent(s). You can NOT use this tool before you have tried your best. + """ + return f"Case not resolved. The desired agent(s) is not created or tested successfully. Details: {task_response}" + agent_creator_agent.functions.extend([case_resolved, case_not_resolved]) + messages.append({"role": "user", "content": f"""\ +The user's request to create agent(s) is: {requirements} +Given the completed agent form with XML format: {output_xml_form} +After previous attempts, you have created new tools that required by the desired agent(s). + +Your task is to create the desired agent(s) for me, note that you may create ONE single agent or multiple agents connected by orchestrator agent. + +After you have created the agent(s), you should test the agent(s) by running the agent(s) using `run_agent` tool to complete the user's task: +{task} + +Note that you can NOT stop util you have created the agent(s) and tested it successfully. +{suggestions} +"""}) + response = client.run(agent_creator_agent, messages, context_variables, debug=debug) + content = response.messages[-1]["content"] + for i in range(MAX_RETRY): + if content.startswith("Case resolved"): + return content, messages + messages.append({"role": "user", "content": f"""\ +The user's request to create agent(s) is: {requirements} +Given the completed agent form with XML format: {output_xml_form} +After previous attempts, you have created new tools that required by the desired agent(s). + +Your task is to create the desired agent(s) for me, note that you may create ONE single agent or multiple agents connected by orchestrator agent. + +After you have created the agent(s), you should test the agent(s) by running the agent(s) using `run_agent` tool to complete the user's task: +{task} + +Note that you can NOT stop util you have created the agent(s) and tested it successfully. +The last attempt failed with the following error: {content}, please try again to create the desired agent(s). +{suggestions} +"""}) + response = client.run(agent_creator_agent, messages, context_variables, debug=debug) + content = response.messages[-1]["content"] + if i == MAX_RETRY: + return f"{content}\nThe desired agent(s) is not created or tested successfully with {MAX_RETRY} attempts.", messages + + +def meta_agent(model: str, context_variables: dict, debug: bool = True): + logger = LoggerManager.get_logger() + # generate agent form + agent_former = get_agent_former_agent(model) + tool_editor_agent = get_tool_editor_agent(model) + agent_creator_agent = get_agent_creator_agent(model) + # enter agent + agent = agent_former + agents = {agent_former.name.replace(' ', '_'): agent_former, tool_editor_agent.name.replace(' ', '_'): tool_editor_agent, agent_creator_agent.name.replace(' ', '_'): agent_creator_agent} + style = Style.from_dict({ + 'bottom-toolbar': 'bg:#333333 #ffffff', + }) + # 创建会话 + session = PromptSession( + completer=UserCompleter(agents.keys()), + complete_while_typing=True, + style=style + ) + client = MetaChain(log_path=logger) + console = Console() + messages = [] + + last_message = "Tell me what do you want to create with `Agent Chain`?" + + while True: + query = session.prompt( + f'{last_message} (type "exit" to quit, press "Enter" to continue): ', + bottom_toolbar=HTML('Prompt: Enter @ to mention Agents'), + ) + if query.strip().lower() == 'exit': + + logo_text = "Agent Chain completed. See you next time! :waving_hand:" + console.print(Panel(logo_text, style="bold salmon1", expand=True)) + break + words = query.split() + console.print(f"[bold green]Your request: {query}[/bold green]", end=" ") + for word in words: + if word.startswith('@') and word[1:] in agents.keys(): + # print(f"[bold magenta]{word}[bold magenta]", end=' ') + agent = agents[word.replace('@', '')] + else: + # print(word, end=' ') + pass + print() + agent_name = agent.name + console.print(f"[bold green][bold magenta]@{agent_name}[/bold magenta] will help you, be patient...[/bold green]") + + match agent_name: + case 'Agent Former Agent': + if query == "": + console.print(f"[bold red]There MUST be a request to create the agent form.[/bold red]") + continue + requirements = query + agent_form, output_xml_form, messages = agent_profiling(agent_former, client, messages, context_variables, requirements, debug) + if agent_form is None: + console.print(f"[bold red][bold magenta]@{agent_name}[/bold magenta] has not created agent form successfully, please modify your requirements again.[/bold red]") + last_message = "Tell me what do you want to create with `Agent Chain`?" + continue + + agent = tool_editor_agent + console.print(f"[bold green][bold magenta]@{agent_name}[/bold magenta] has created agent form successfully with the following details:\n[/bold green][bold blue]{output_xml_form}[/bold blue]") + last_message = "It is time to create the desired tools, do you have any suggestions for creating the tools?" + case 'Tool Editor Agent': + suggestions = query + tool_response, messages = tool_editing(tool_editor_agent, client, messages, context_variables, agent_form, output_xml_form, debug, suggestions) + if tool_response.startswith("Case not resolved"): + console.print(f"[bold red][bold magenta]@{agent_name}[/bold magenta] has not created tools successfully with the following error: {tool_response}[/bold red]") + agent = tool_editor_agent + last_message = "The tools are not created successfully, do you have any suggestions for creating the tools?" + continue + elif tool_response.startswith("Case resolved"): + agent = agent_creator_agent + console.print(f"[bold green][bold magenta]@{agent_name}[/bold magenta] has created tools successfully with the following details:\n[/bold green][bold blue]{tool_response}[/bold blue]") + last_message = "It is time to create the desired agent(s), do you have any suggestions for creating the agent(s)?" + else: + raise ValueError(f"Unknown tool response: {tool_response}") + + case 'Agent Creator Agent': + suggestions = query + default_value='Come up with a task for the agent(s) to test your created agent(s), and use `run_agent` tool to test your created agent(s).' # 这里设置你想要的默认值 + task = session.prompt( + 'It is time to create the desired agent(s), what task do you want to complete with the agent(s)? (Press Enter if none): ', + + ) + task = default_value if not task.strip() else task + agent_response, messages = agent_editing(agent_creator_agent, client, messages, context_variables, agent_form, output_xml_form, requirements, task, debug, suggestions) + if agent_response.startswith("Case not resolved"): + console.print(f"[bold red][bold magenta]@{agent_name}[/bold magenta] has not created agent(s) successfully with the following error: {agent_response}[/bold red]") + agent = agent_creator_agent + last_message = "The agent(s) are not created successfully, do you have any suggestions for creating the agent(s)?" + continue + else: + console.print(f"[bold green][bold magenta]@{agent_name}[/bold magenta] has created agent(s) successfully with the following details:\n[/bold green][bold blue]{agent_response}[/bold blue]") + last_message = "Tell me what do you want to create with `Agent Chain`?" diff --git a/autoagent/cli_utils/metachain_meta_workflow.py b/autoagent/cli_utils/metachain_meta_workflow.py new file mode 100644 index 0000000..98324bc --- /dev/null +++ b/autoagent/cli_utils/metachain_meta_workflow.py @@ -0,0 +1,194 @@ +from autoagent import MetaChain +from autoagent.util import ask_text, single_select_menu, print_markdown, debug_print, UserCompleter +from prompt_toolkit import PromptSession +from prompt_toolkit.formatted_text import HTML +from prompt_toolkit.styles import Style +from autoagent.logger import LoggerManager, MetaChainLogger +from rich.console import Console +from rich.panel import Panel +from autoagent.agents.meta_agent.workflow_former import get_workflow_former_agent +from autoagent.agents.meta_agent.workflow_creator import get_workflow_creator_agent +import re +from autoagent.agents.meta_agent.worklow_form_complie import parse_workflow_form, WorkflowForm + +def workflow_profiling(workflow_former, client, messages, context_variables, requirements, debug): + messages.append({"role": "user", "content": requirements + """ +Directly output the form in the XML format without ANY other text. +"""}) + response = client.run(workflow_former, messages, context_variables, debug=debug) + output_xml_form = response.messages[-1]["content"] + messages.extend(response.messages) + + MAX_RETRY = 3 + for i in range(MAX_RETRY): + workflow_form = parse_workflow_form(output_xml_form) + if isinstance(workflow_form, WorkflowForm): + break + elif isinstance(workflow_form, str): + print(f"Error parsing XML to workflow form: {workflow_form}. Retry {i+1}/{MAX_RETRY}") + messages.append({"role": "user", "content": f"Error parsing XML to workflow form, the error message is: {workflow_form}\nNote that there are some special restrictions for creating workflow form, please try again."}) + response = client.run(workflow_former, messages, context_variables, debug=debug) + output_xml_form = response.messages[-1]["content"] + messages.extend(response.messages) + else: + raise ValueError(f"Unexpected error: {workflow_form}") + return workflow_form, output_xml_form, messages + +def workflow_editing(workflow_creator_agent, client, messages, context_variables, workflow_form, output_xml_form, requirements, task, debug, suggestions = ""): + MAX_RETRY = 3 + if suggestions != "": + suggestions = "[IMPORTANT] Here are some suggestions for creating the workflow: " + suggestions + agents = workflow_form.agents + new_agents = [] + for agent in agents: + if agent.category == "new": + new_agents.append(agent) + + if len(new_agents) != 0: + new_agent_str = "AGENT CREATION INSTRUCTIONS:\nBefore you create the workflow, you need to create the following new agents in the workflow:\n" + for agent in new_agents: + new_agent_str += f"Agent name: {agent.name}\nAgent description: {agent.description}\n" + new_agent_str += f"Agent tools: {agent.tools}\n" if agent.tools else "Agent tools: []\n" + else: + new_agent_str = "" + + def case_resolved(task_response: str, context_variables: dict): + """ + Use this tools when the desired workflow is created and tested successfully. You can NOT use this tool if the workflow is not created or tested successfully by running the workflow. + """ + return f"Case resolved. The desired workflow is created and tested successfully. : {task_response}" + def case_not_resolved(task_response: str, context_variables: dict): + """ + Use this tools when you encounter irresistible errors after trying your best with multiple attempts for creating the desired workflow. You can NOT use this tool before you have tried your best. + """ + return f"Case not resolved. The desired workflow is not created or tested successfully. Details: {task_response}" + workflow_creator_agent.functions.extend([case_resolved, case_not_resolved]) + messages.append({"role": "user", "content": f"""\ +WORKFLOW CREATION INSTRUCTIONS: +The user's request to create workflow is: {requirements} +Given the completed workflow form with XML format: {output_xml_form} + +TASK: +Your task is to create the workflow for me, and then test the workflow by running the workflow using `run_workflow` tool to complete the user's task: +{task} + +{new_agent_str} + +TERMINATION INSTRUCTIONS: +After you have created the workflow and tested it successfully, you can use the `case_resolved` tool to indicate the case is resolved, otherwise you should try your best to create the workflow. And ONLY after you have tried multiple times, you can use the `case_not_resolved` tool to indicate the case is not resolved and give the reason. + +Remember: you can NOT stop util you have created the workflow and tested it successfully. +{suggestions} +"""}) + response = client.run(workflow_creator_agent, messages, context_variables, debug=debug) + content = response.messages[-1]["content"] + for i in range(MAX_RETRY): + if content.startswith("Case resolved"): + return content, messages + messages.append({"role": "user", "content": f"""\ +WORKFLOW CREATION INSTRUCTIONS: +The user's request to create workflow is: {requirements} +Given the completed workflow form with XML format: {output_xml_form} + +TASK: +Your task is to create the workflow for me, and then test the workflow by running the workflow using `run_workflow` tool to complete the user's task: +{task} + +{new_agent_str} + +TERMINATION INSTRUCTIONS: +After you have created the workflow and tested it successfully, you can use the `case_resolved` tool to indicate the case is resolved, otherwise you should try your best to create the workflow. And ONLY after you have tried multiple times, you can use the `case_not_resolved` tool to indicate the case is not resolved and give the reason. + +Remember: you can NOT stop util you have created the workflow and tested it successfully. + +FEEDBACK: +The last attempt failed with the following error: {content}, please try again to create the desired workflow. +{suggestions} +"""}) + response = client.run(workflow_creator_agent, messages, context_variables, debug=debug) + content = response.messages[-1]["content"] + if i == MAX_RETRY: + return f"The desired workflow is not created or tested successfully with {MAX_RETRY} attempts.", messages + + + + +def meta_workflow(model: str, context_variables: dict, debug: bool = True): + print('\033[s\033[?25l', end='') # Save cursor position and hide cursor + logger = LoggerManager.get_logger() + workflow_former = get_workflow_former_agent(model) + workflow_creator_agent = get_workflow_creator_agent(model) + + agent = workflow_former + agents = {workflow_former.name.replace(' ', '_'): workflow_former, workflow_creator_agent.name.replace(' ', '_'): workflow_creator_agent} + style = Style.from_dict({ + 'bottom-toolbar': 'bg:#333333 #ffffff', + }) + # 创建会话 + session = PromptSession( + completer=UserCompleter(agents.keys()), + complete_while_typing=True, + style=style + ) + + client = MetaChain(log_path=logger) + console = Console() + messages = [] + + last_message = "Tell me what do you want to create with `Workflow Chain`?" + + while True: + query = session.prompt( + f'{last_message} (type "exit" to quit, press "Enter" to continue): ', + bottom_toolbar=HTML('Prompt: Enter @ to mention Agents'), + ) + if query.strip().lower() == 'exit': + + logo_text = "Workflow Chain completed. See you next time! :waving_hand:" + console.print(Panel(logo_text, style="bold salmon1", expand=True)) + break + words = query.split() + console.print(f"[bold green]Your request: {query}[/bold green]", end=" ") + for word in words: + if word.startswith('@') and word[1:] in agents.keys(): + # print(f"[bold magenta]{word}[bold magenta]", end=' ') + agent = agents[word.replace('@', '')] + else: + # print(word, end=' ') + pass + print() + agent_name = agent.name + console.print(f"[bold green][bold magenta]@{agent_name}[/bold magenta] will help you, be patient...[/bold green]") + match agent_name: + case "Workflow Former Agent": + if query == "": + console.print(f"[bold red]There MUST be a request to create the agent form.[/bold red]") + continue + requirements = query + workflow_form, output_xml_form, messages = workflow_profiling(workflow_former, client, messages, context_variables, requirements, debug) + if workflow_form is None: + console.print(f"[bold red][bold magenta]@{agent_name}[/bold magenta] has not created workflow form successfully, please modify your requirements again.[/bold red]") + last_message = "Tell me what do you want to create with `Workflow Chain`?" + continue + agent = workflow_creator_agent + context_variables["workflow_form"] = workflow_form + console.print(f"[bold green][bold magenta]@{agent_name}[/bold magenta] has created workflow form successfully with the following details:\n[/bold green][bold blue]{output_xml_form}[/bold blue]") + last_message = "It is time to create the desired workflow, do you have any suggestions for creating the workflow?" + case "Workflow Creator Agent": + suggestions = query + default_value='Come up with a task for the workflow to test your created workflow, and use `run_workflow` tool to test your created workflow.' # 这里设置你想要的默认值 + task = session.prompt( + 'It is time to create the desired workflow, what task do you want to complete with the workflow? (Press Enter if none): ', + + ) + task = default_value if not task.strip() else task + agent_response, messages = workflow_editing(workflow_creator_agent, client, messages, context_variables, workflow_form, output_xml_form, requirements, task, debug, suggestions) + if agent_response.startswith("Case not resolved"): + console.print(f"[bold red][bold magenta]@{agent_name}[/bold magenta] has not created workflow successfully with the following error: {agent_response}[/bold red]") + agent = workflow_creator_agent + else: + console.print(f"[bold green][bold magenta]@{agent_name}[/bold magenta] has created workflow successfully with the following details:\n[/bold green][bold blue]{agent_response}[/bold blue]") + last_message = "Tell me what do you want to create with `Workflow Chain` next?" + + + \ No newline at end of file diff --git a/autoagent/core.py b/autoagent/core.py new file mode 100644 index 0000000..9aacbd3 --- /dev/null +++ b/autoagent/core.py @@ -0,0 +1,673 @@ +# Standard library imports +import copy +import json +from collections import defaultdict +from typing import List, Callable, Union +from datetime import datetime +# Local imports +import os +from .util import function_to_json, debug_print, merge_chunk, pretty_print_messages +from .types import ( + Agent, + AgentFunction, + Message, + ChatCompletionMessageToolCall, + Function, + Response, + Result, +) +from litellm import completion, acompletion +from pathlib import Path +from .logger import MetaChainLogger, LoggerManager +from httpx import RemoteProtocolError, ConnectError +from litellm.exceptions import APIError +from tenacity import ( + retry, + stop_after_attempt, + wait_exponential, + retry_if_exception_type +) +from openai import AsyncOpenAI +import litellm +import inspect +from constant import MC_MODE, FN_CALL, API_BASE_URL, NOT_SUPPORT_SENDER, ADD_USER, NON_FN_CALL +from autoagent.fn_call_converter import convert_tools_to_description, convert_non_fncall_messages_to_fncall_messages, SYSTEM_PROMPT_SUFFIX_TEMPLATE, convert_fn_messages_to_non_fn_messages, interleave_user_into_messages +from litellm.types.utils import Message as litellmMessage +# litellm.set_verbose=True +# client = AsyncOpenAI() +def should_retry_error(exception): + if MC_MODE is False: print(f"Caught exception: {type(exception).__name__} - {str(exception)}") + + # 匹配更多错误类型 + if isinstance(exception, (APIError, RemoteProtocolError, ConnectError)): + return True + + # 通过错误消息匹配 + error_msg = str(exception).lower() + return any([ + "connection error" in error_msg, + "server disconnected" in error_msg, + "eof occurred" in error_msg, + "timeout" in error_msg, + "event loop is closed" in error_msg, # 添加事件循环错误 + "anthropicexception" in error_msg, # 添加 Anthropic 相关错误 + ]) +__CTX_VARS_NAME__ = "context_variables" +logger = LoggerManager.get_logger() + +def adapt_tools_for_gemini(tools): + """为 Gemini 模型适配工具定义，确保所有 OBJECT 类型参数都有非空的 properties""" + if tools is None: + return None + + adapted_tools = [] + for tool in tools: + adapted_tool = copy.deepcopy(tool) + + # 检查参数 + if "parameters" in adapted_tool["function"]: + params = adapted_tool["function"]["parameters"] + + # 处理顶层参数 + if params.get("type") == "object": + if "properties" not in params or not params["properties"]: + params["properties"] = { + "dummy": { + "type": "string", + "description": "Dummy property for Gemini compatibility" + } + } + + # 处理嵌套参数 + if "properties" in params: + for prop_name, prop in params["properties"].items(): + if isinstance(prop, dict) and prop.get("type") == "object": + if "properties" not in prop or not prop["properties"]: + prop["properties"] = { + "dummy": { + "type": "string", + "description": "Dummy property for Gemini compatibility" + } + } + + adapted_tools.append(adapted_tool) + return adapted_tools + +class MetaChain: + def __init__(self, log_path: Union[str, None, MetaChainLogger] = None): + """ + log_path: path of log file, None + """ + if logger: + self.logger = logger + elif isinstance(log_path, MetaChainLogger): + self.logger = log_path + else: + self.logger = MetaChainLogger(log_path=log_path) + # if self.logger.log_path is None: self.logger.info("[Warning] Not specific log path, so log will not be saved", "...", title="Log Path", color="light_cyan3") + # else: self.logger.info("Log file is saved to", self.logger.log_path, "...", title="Log Path", color="light_cyan3") + # @retry( + # stop=stop_after_attempt(4), + # wait=wait_exponential(multiplier=1, min=4, max=60), + # retry=should_retry_error, + # before_sleep=lambda retry_state: print(f"Retrying... (attempt {retry_state.attempt_number})") + # ) + def get_chat_completion( + self, + agent: Agent, + history: List, + context_variables: dict, + model_override: str, + stream: bool, + debug: bool, + ) -> Message: + context_variables = defaultdict(str, context_variables) + instructions = ( + agent.instructions(context_variables) + if callable(agent.instructions) + else agent.instructions + ) + if agent.examples: + examples = agent.examples(context_variables) if callable(agent.examples) else agent.examples + history = examples + history + + messages = [{"role": "system", "content": instructions}] + history + # debug_print(debug, "Getting chat completion for...:", messages) + + tools = [function_to_json(f) for f in agent.functions] + # hide context_variables from model + for tool in tools: + params = tool["function"]["parameters"] + params["properties"].pop(__CTX_VARS_NAME__, None) + if __CTX_VARS_NAME__ in params["required"]: + params["required"].remove(__CTX_VARS_NAME__) + create_model = model_override or agent.model + + if "gemini" in create_model.lower(): + tools = adapt_tools_for_gemini(tools) + if FN_CALL: + # create_model = model_override or agent.model + assert litellm.supports_function_calling(model = create_model) == True, f"Model {create_model} does not support function calling, please set `FN_CALL=False` to use non-function calling mode" + create_params = { + "model": create_model, + "messages": messages, + "tools": tools or None, + "tool_choice": agent.tool_choice, + "stream": stream, + } + NO_SENDER_MODE = False + for not_sender_model in NOT_SUPPORT_SENDER: + if not_sender_model in create_model: + NO_SENDER_MODE = True + break + + if NO_SENDER_MODE: + messages = create_params["messages"] + for message in messages: + if 'sender' in message: + del message['sender'] + create_params["messages"] = messages + + if tools and create_params['model'].startswith("gpt"): + create_params["parallel_tool_calls"] = agent.parallel_tool_calls + completion_response = completion(**create_params) + else: + # create_model = model_override or agent.model + assert agent.tool_choice == "required", f"Non-function calling mode MUST use tool_choice = 'required' rather than {agent.tool_choice}" + last_content = messages[-1]["content"] + tools_description = convert_tools_to_description(tools) + messages[-1]["content"] = last_content + "\n[IMPORTANT] You MUST use the tools provided to complete the task.\n" + SYSTEM_PROMPT_SUFFIX_TEMPLATE.format(description=tools_description) + NO_SENDER_MODE = False + for not_sender_model in NOT_SUPPORT_SENDER: + if not_sender_model in create_model: + NO_SENDER_MODE = True + break + + if NO_SENDER_MODE: + for message in messages: + if 'sender' in message: + del message['sender'] + if NON_FN_CALL: + messages = convert_fn_messages_to_non_fn_messages(messages) + if ADD_USER and messages[-1]["role"] != "user": + # messages.append({"role": "user", "content": "Please think twice and take the next action according to your previous actions and observations."}) + messages = interleave_user_into_messages(messages) + create_params = { + "model": create_model, + "messages": messages, + "stream": stream, + "base_url": API_BASE_URL, + } + completion_response = completion(**create_params) + last_message = [{"role": "assistant", "content": completion_response.choices[0].message.content}] + converted_message = convert_non_fncall_messages_to_fncall_messages(last_message, tools) + if "tool_calls" in converted_message[0]: + converted_tool_calls = [ChatCompletionMessageToolCall(**tool_call) for tool_call in converted_message[0]["tool_calls"]] + else: + converted_tool_calls = None + completion_response.choices[0].message = litellmMessage(content = converted_message[0]["content"], role = "assistant", tool_calls = converted_tool_calls) + + return completion_response + + def handle_function_result(self, result, debug) -> Result: + match result: + case Result() as result: + return result + + case Agent() as agent: + return Result( + value=json.dumps({"assistant": agent.name}), + agent=agent, + ) + case _: + try: + return Result(value=str(result)) + except Exception as e: + error_message = f"Failed to cast response to string: {result}. Make sure agent functions return a string or Result object. Error: {str(e)}" + self.logger.info(error_message, title="Handle Function Result Error", color="red") + raise TypeError(error_message) + + def handle_tool_calls( + self, + tool_calls: List[ChatCompletionMessageToolCall], + functions: List[AgentFunction], + context_variables: dict, + debug: bool, + handle_mm_func: Callable[[], str] = None, + ) -> Response: + function_map = {f.__name__: f for f in functions} + partial_response = Response( + messages=[], agent=None, context_variables={}) + + for tool_call in tool_calls: + name = tool_call.function.name + # handle missing tool case, skip to next tool + if name not in function_map: + self.logger.info(f"Tool {name} not found in function map. You are recommended to use `run_tool` to run this tool.", title="Tool Call Error", color="red") + partial_response.messages.append( + { + "role": "tool", + "tool_call_id": tool_call.id, + "name": name, + "content": f"Error: Tool {name} not found. You are recommended to use `run_tool` to run this tool.", + } + ) + continue + args = json.loads(tool_call.function.arguments) + + # debug_print( + # debug, f"Processing tool call: {name} with arguments {args}") + func = function_map[name] + # pass context_variables to agent functions + # if __CTX_VARS_NAME__ in func.__code__.co_varnames: + # args[__CTX_VARS_NAME__] = context_variables + if __CTX_VARS_NAME__ in inspect.signature(func).parameters.keys(): + args[__CTX_VARS_NAME__] = context_variables + raw_result = function_map[name](**args) + + result: Result = self.handle_function_result(raw_result, debug) + + partial_response.messages.append( + { + "role": "tool", + "tool_call_id": tool_call.id, + "name": name, + "content": result.value, + } + ) + self.logger.pretty_print_messages(partial_response.messages[-1]) + if result.image: + assert handle_mm_func, f"handle_mm_func is not provided, but an image is returned by tool call {name}({tool_call.function.arguments})" + partial_response.messages.append( + { + "role": "user", + "content": [ + # {"type":"text", "text":f"After take last action `{name}({tool_call.function.arguments})`, the image of current page is shown below. Please take next action based on the image, the current state of the page as well as previous actions and observations."}, + {"type":"text", "text":handle_mm_func(name, tool_call.function.arguments)}, + { + "type":"image_url", + "image_url":{ + "url":f"data:image/png;base64,{result.image}" + } + } + ] + } + ) + # debug_print(debug, "Tool calling: ", json.dumps(partial_response.messages[-1], indent=4), log_path=log_path, title="Tool Calling", color="green") + + partial_response.context_variables.update(result.context_variables) + if result.agent: + partial_response.agent = result.agent + + return partial_response + + def run_and_stream( + self, + agent: Agent, + messages: List, + context_variables: dict = {}, + model_override: str = None, + debug: bool = False, + max_turns: int = float("inf"), + execute_tools: bool = True, + ): + active_agent = agent + context_variables = copy.deepcopy(context_variables) + history = copy.deepcopy(messages) + init_len = len(messages) + + while len(history) - init_len < max_turns: + + message = { + "content": "", + "sender": agent.name, + "role": "assistant", + "function_call": None, + "tool_calls": defaultdict( + lambda: { + "function": {"arguments": "", "name": ""}, + "id": "", + "type": "", + } + ), + } + + # get completion with current history, agent + completion = self.get_chat_completion( + agent=active_agent, + history=history, + context_variables=context_variables, + model_override=model_override, + stream=True, + debug=debug, + ) + + yield {"delim": "start"} + for chunk in completion: + delta = json.loads(chunk.choices[0].delta.json()) + if delta["role"] == "assistant": + delta["sender"] = active_agent.name + yield delta + delta.pop("role", None) + delta.pop("sender", None) + merge_chunk(message, delta) + yield {"delim": "end"} + + message["tool_calls"] = list( + message.get("tool_calls", {}).values()) + if not message["tool_calls"]: + message["tool_calls"] = None + debug_print(debug, "Received completion:", message) + history.append(message) + + if not message["tool_calls"] or not execute_tools: + debug_print(debug, "Ending turn.") + break + + # convert tool_calls to objects + tool_calls = [] + for tool_call in message["tool_calls"]: + function = Function( + arguments=tool_call["function"]["arguments"], + name=tool_call["function"]["name"], + ) + tool_call_object = ChatCompletionMessageToolCall( + id=tool_call["id"], function=function, type=tool_call["type"] + ) + tool_calls.append(tool_call_object) + + # handle function calls, updating context_variables, and switching agents + partial_response = self.handle_tool_calls( + tool_calls, active_agent.functions, context_variables, debug + ) + history.extend(partial_response.messages) + context_variables.update(partial_response.context_variables) + if partial_response.agent: + active_agent = partial_response.agent + + yield { + "response": Response( + messages=history[init_len:], + agent=active_agent, + context_variables=context_variables, + ) + } + + def run( + self, + agent: Agent, + messages: List, + context_variables: dict = {}, + model_override: str = None, + stream: bool = False, + debug: bool = True, + max_turns: int = float("inf"), + execute_tools: bool = True, + ) -> Response: + if stream: + return self.run_and_stream( + agent=agent, + messages=messages, + context_variables=context_variables, + model_override=model_override, + debug=debug, + max_turns=max_turns, + execute_tools=execute_tools, + ) + active_agent = agent + enter_agent = agent + context_variables = copy.copy(context_variables) + history = copy.deepcopy(messages) + init_len = len(messages) + + self.logger.info("Receiveing the task:", history[-1]['content'], title="Receive Task", color="green") + + while len(history) - init_len < max_turns and active_agent: + + # get completion with current history, agent + completion = self.get_chat_completion( + agent=active_agent, + history=history, + context_variables=context_variables, + model_override=model_override, + stream=stream, + debug=debug, + ) + message: Message = completion.choices[0].message + message.sender = active_agent.name + # debug_print(debug, "Received completion:", message.model_dump_json(indent=4), log_path=log_path, title="Received Completion", color="blue") + self.logger.pretty_print_messages(message) + history.append( + json.loads(message.model_dump_json()) + ) # to avoid OpenAI types (?) + + # if not message.tool_calls or not execute_tools: + # self.logger.info("Ending turn.", title="End Turn", color="red") + # break + + if enter_agent.tool_choice != "required": + if (not message.tool_calls and active_agent.name == enter_agent.name) or not execute_tools: + self.logger.info("Ending turn.", title="End Turn", color="red") + break + else: + if (message.tool_calls and message.tool_calls[0].function.name == "case_resolved") or not execute_tools: + self.logger.info("Ending turn with case resolved.", title="End Turn", color="red") + partial_response = self.handle_tool_calls( + message.tool_calls, active_agent.functions, context_variables, debug, handle_mm_func=active_agent.handle_mm_func + ) + history.extend(partial_response.messages) + context_variables.update(partial_response.context_variables) + break + elif (message.tool_calls and message.tool_calls[0].function.name == "case_not_resolved") or not execute_tools: + self.logger.info("Ending turn with case not resolved.", title="End Turn", color="red") + partial_response = self.handle_tool_calls( + message.tool_calls, active_agent.functions, context_variables, debug, handle_mm_func=active_agent.handle_mm_func + ) + history.extend(partial_response.messages) + context_variables.update(partial_response.context_variables) + break + elif (not message.tool_calls) or not execute_tools: + self.logger.info("Ending turn with no tool calls.", title="End Turn", color="red") + break + + # if (message.tool_calls and message.tool_calls[0].function.name == "case_resolved") or not execute_tools: + # debug_print(debug, "Ending turn.", log_path=log_path, title="End Turn", color="red") + # break + + # handle function calls, updating context_variables, and switching agents + if message.tool_calls: + partial_response = self.handle_tool_calls( + message.tool_calls, active_agent.functions, context_variables, debug, handle_mm_func=active_agent.handle_mm_func + ) + else: + partial_response = Response(messages=[message]) + history.extend(partial_response.messages) + context_variables.update(partial_response.context_variables) + if partial_response.agent: + active_agent = partial_response.agent + + return Response( + messages=history[init_len:], + agent=active_agent, + context_variables=context_variables, + ) + @retry( + stop=stop_after_attempt(4), + wait=wait_exponential(multiplier=1, min=10, max=180), + retry=should_retry_error, + before_sleep=lambda retry_state: print(f"Retrying... (attempt {retry_state.attempt_number})") + ) + async def get_chat_completion_async( + self, + agent: Agent, + history: List, + context_variables: dict, + model_override: str, + stream: bool, + debug: bool, + ) -> Message: + context_variables = defaultdict(str, context_variables) + instructions = ( + agent.instructions(context_variables) + if callable(agent.instructions) + else agent.instructions + ) + if agent.examples: + examples = agent.examples(context_variables) if callable(agent.examples) else agent.examples + history = examples + history + + messages = [{"role": "system", "content": instructions}] + history + # debug_print(debug, "Getting chat completion for...:", messages) + + tools = [function_to_json(f) for f in agent.functions] + # hide context_variables from model + for tool in tools: + params = tool["function"]["parameters"] + params["properties"].pop(__CTX_VARS_NAME__, None) + if __CTX_VARS_NAME__ in params["required"]: + params["required"].remove(__CTX_VARS_NAME__) + + if FN_CALL: + create_model = model_override or agent.model + assert litellm.supports_function_calling(model = create_model) == True, f"Model {create_model} does not support function calling, please set `FN_CALL=False` to use non-function calling mode" + + create_params = { + "model": create_model, + "messages": messages, + "tools": tools or None, + "tool_choice": agent.tool_choice, + "stream": stream, + } + NO_SENDER_MODE = False + for not_sender_model in NOT_SUPPORT_SENDER: + if not_sender_model in create_model: + NO_SENDER_MODE = True + break + + if NO_SENDER_MODE: + messages = create_params["messages"] + for message in messages: + if 'sender' in message: + del message['sender'] + create_params["messages"] = messages + + if tools and create_params['model'].startswith("gpt"): + create_params["parallel_tool_calls"] = agent.parallel_tool_calls + completion_response = await acompletion(**create_params) + else: + create_model = model_override or agent.model + assert agent.tool_choice == "required", f"Non-function calling mode MUST use tool_choice = 'required' rather than {agent.tool_choice}" + last_content = messages[-1]["content"] + tools_description = convert_tools_to_description(tools) + messages[-1]["content"] = last_content + "\n[IMPORTANT] You MUST use the tools provided to complete the task.\n" + SYSTEM_PROMPT_SUFFIX_TEMPLATE.format(description=tools_description) + NO_SENDER_MODE = False + for not_sender_model in NOT_SUPPORT_SENDER: + if not_sender_model in create_model: + NO_SENDER_MODE = True + break + + if NO_SENDER_MODE: + for message in messages: + if 'sender' in message: + del message['sender'] + create_params = { + "model": create_model, + "messages": messages, + "stream": stream, + "base_url": API_BASE_URL, + } + completion_response = await acompletion(**create_params) + last_message = [{"role": "assistant", "content": completion_response.choices[0].message.content}] + converted_message = convert_non_fncall_messages_to_fncall_messages(last_message, tools) + converted_tool_calls = [ChatCompletionMessageToolCall(**tool_call) for tool_call in converted_message[0]["tool_calls"]] + completion_response.choices[0].message = litellmMessage(content = converted_message[0]["content"], role = "assistant", tool_calls = converted_tool_calls) + + # response = await acompletion(**create_params) + # response = await client.chat.completions.create(**create_params) + return completion_response + + async def run_async( + self, + agent: Agent, + messages: List, + context_variables: dict = {}, + model_override: str = None, + stream: bool = False, + debug: bool = True, + max_turns: int = float("inf"), + execute_tools: bool = True, + ) -> Response: + assert stream == False, "Async run does not support stream" + active_agent = agent + enter_agent = agent + context_variables = copy.copy(context_variables) + history = copy.deepcopy(messages) + init_len = len(messages) + + self.logger.info("Receiveing the task:", history[-1]['content'], title="Receive Task", color="green") + + while len(history) - init_len < max_turns and active_agent: + + # get completion with current history, agent + completion = await self.get_chat_completion_async( + agent=active_agent, + history=history, + context_variables=context_variables, + model_override=model_override, + stream=stream, + debug=debug, + ) + message: Message = completion.choices[0].message + message.sender = active_agent.name + # debug_print(debug, "Received completion:", message.model_dump_json(indent=4), log_path=log_path, title="Received Completion", color="blue") + self.logger.pretty_print_messages(message) + history.append( + json.loads(message.model_dump_json()) + ) # to avoid OpenAI types (?) + + if enter_agent.tool_choice != "required": + if (not message.tool_calls and active_agent.name == enter_agent.name) or not execute_tools: + self.logger.info("Ending turn.", title="End Turn", color="red") + break + else: + if (message.tool_calls and message.tool_calls[0].function.name == "case_resolved") or not execute_tools: + self.logger.info("Ending turn with case resolved.", title="End Turn", color="red") + partial_response = self.handle_tool_calls( + message.tool_calls, active_agent.functions, context_variables, debug, handle_mm_func=active_agent.handle_mm_func + ) + history.extend(partial_response.messages) + context_variables.update(partial_response.context_variables) + break + elif (message.tool_calls and message.tool_calls[0].function.name == "case_not_resolved") or not execute_tools: + self.logger.info("Ending turn with case not resolved.", title="End Turn", color="red") + partial_response = self.handle_tool_calls( + message.tool_calls, active_agent.functions, context_variables, debug, handle_mm_func=active_agent.handle_mm_func + ) + history.extend(partial_response.messages) + context_variables.update(partial_response.context_variables) + break + elif (not message.tool_calls) or not execute_tools: + self.logger.info("Ending turn with no tool calls.", title="End Turn", color="red") + break + + # if (message.tool_calls and message.tool_calls[0].function.name == "case_resolved") or not execute_tools: + # debug_print(debug, "Ending turn.", log_path=log_path, title="End Turn", color="red") + # break + + # handle function calls, updating context_variables, and switching agents + if message.tool_calls: + partial_response = self.handle_tool_calls( + message.tool_calls, active_agent.functions, context_variables, debug, handle_mm_func=active_agent.handle_mm_func + ) + else: + partial_response = Response(messages=[message]) + history.extend(partial_response.messages) + context_variables.update(partial_response.context_variables) + if partial_response.agent: + active_agent = partial_response.agent + + return Response( + messages=history[init_len:], + agent=active_agent, + context_variables=context_variables, + ) diff --git a/autoagent/environment/__init__.py b/autoagent/environment/__init__.py new file mode 100644 index 0000000..359d4e3 --- /dev/null +++ b/autoagent/environment/__init__.py @@ -0,0 +1,5 @@ +from .docker_env import DockerEnv, DockerConfig +from .local_env import LocalEnv +from .browser_env import BrowserEnv, VIEWPORT +from .markdown_browser import RequestsMarkdownBrowser +from .utils import setup_metachain \ No newline at end of file diff --git a/autoagent/environment/browser_cookies.py b/autoagent/environment/browser_cookies.py new file mode 100644 index 0000000..2437f6c --- /dev/null +++ b/autoagent/environment/browser_cookies.py @@ -0,0 +1,42 @@ +import json +from pathlib import Path +import glob + +wd = Path(__file__).parent.resolve() +def load_cookies_from_json(json_path): + with open(json_path, 'r') as f: + cookies = json.load(f) + return cookies + + +def convert_cookies_to_python(): + all_cookies = [] + # cookie_files = [ + # "orcid.org.cookies.json", + # "www.researchgate.net.cookies.json", + # "github.com.cookies.json", + # "www.youtube.com.cookies.json", + # "www.ncbi.nlm.nih.gov.cookies.json", + # "archive.org.cookies.json", + # "nature.com.cookies.json" + # ] + json_dir = wd / "cookie_json" + cookie_files = glob.glob(str(json_dir / "*.json")) + + for cookie_file in cookie_files: + json_path = wd / "cookie_json" / cookie_file + cookies = load_cookies_from_json(json_path) + all_cookies.extend(cookies) + + # 生成Python格式的cookies文件 + output_path = wd / "cookies_data.py" + output_str = "COOKIES_LIST = [\n" + for cookie in all_cookies: + output_str += f" {repr(cookie)},\n" + output_str += "]\n" + with open(output_path, "w", encoding="utf-8") as f: + f.write(output_str) + return output_str + +if __name__ == "__main__": + print(convert_cookies_to_python()) \ No newline at end of file diff --git a/autoagent/environment/browser_env.py b/autoagent/environment/browser_env.py new file mode 100644 index 0000000..45cefb0 --- /dev/null +++ b/autoagent/environment/browser_env.py @@ -0,0 +1,649 @@ +import atexit +import base64 +import io +import json +import multiprocessing +import time +import uuid + +import browsergym.core # noqa F401 (we register the openended task as a gym environment) +import gymnasium as gym +import html2text +import numpy as np +import tenacity +from browsergym.utils.obs import flatten_dom_to_str +from PIL import Image +from autoagent.util import debug_print +from autoagent.logger import LoggerManager +import inspect +import textwrap + +from .shutdown_listener import should_continue, should_exit +from .tenacity_stop import stop_if_should_exit +from datetime import datetime +from pathlib import Path +from browsergym.core.action.functions import goto, page, get_elem_by_bid, demo_mode, tab_focus +import os +from typing import Dict, Union, cast, Literal +from playwright.sync_api import Page, Download +from autoagent.io_utils import read_file +from autoagent.environment.mdconvert import _get_page_markdown +from autoagent.environment.browser_cookies import convert_cookies_to_python +from autoagent.environment.cookies_data import COOKIES_LIST +# from constant import DOCKER_WORKPLACE_NAME, LOCAL_ROOT +from functools import update_wrapper +from inspect import signature +import types +import sys +import tempfile +VIEWPORT = {"width": 1280, "height": 720} + +BROWSER_EVAL_GET_GOAL_ACTION = 'GET_EVAL_GOAL' +BROWSER_EVAL_GET_REWARDS_ACTION = 'GET_EVAL_REWARDS' +class BrowserInitException(Exception): + def __init__(self, message='Failed to initialize browser environment'): + super().__init__(message) +def _local_to_docker(local_path: str): + """ + Convert a local path to a docker path + local_path: the local path to convert, like `{local_workplace}/downloads/xxx` + docker_path: the docker path to convert, like `{docker_workplace}/downloads/xxx` + + Examples: + _local_to_docker('/Users/tangjiabin/Documents/reasoning/autoagent/workplace_gaia_eval/downloads/xxx') + """ + local_workplace = None + docker_workplace = None + assert local_workplace in local_path, f"local_path must contain {local_workplace}" + return local_path.replace(local_workplace, docker_workplace) +def _visit_page(url: str): + """ + Visit a page, including downloading files based on the url + + Examples: + _visit_page('https://archive.org/download/higpt_stage2/instruct_ds_dblp.tar.gz') + """ + # def _local_to_docker(local_path: str): + # """ + # Convert a local path to a docker path + # local_path: the local path to convert, like `{LOCAL_ROOT}/{DOCKER_WORKPLACE_NAME}/downloads/xxx` + # docker_path: the docker path to convert, like `/{DOCKER_WORKPLACE_NAME}/downloads/xxx` + # """ + # assert LOCAL_ROOT in local_path, f"local_path must contain {LOCAL_ROOT}" + # return local_path.replace(LOCAL_ROOT, '') + try: + # 尝试作为普通网页访问 + page.context.add_cookies(COOKIES_LIST) + # goto(url) + page.set_extra_http_headers({ + "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36", + "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8", + "Accept-Language": "en-US,en;q=0.9" + }) + page.goto(url, timeout=6000) + if page.get_by_text("Verify you are human by completing the action below.").count() > 0: + _checkMeetChallenge() + # 等待页面完全加载 + # 增加等待时间，确保页面完全加载 + page.wait_for_load_state("networkidle", timeout=3000) + # page.wait_for_timeout(3000) + + except Exception as e_outer: + # 处理文件下载情况 + if "net::ERR_ABORTED" in str(e_outer): + import os + import requests + import base64 + downloads_folder = f"{local_workplace}/downloads" + + os.makedirs(downloads_folder, exist_ok=True) + filename = os.path.basename(url) + filepath = os.path.join(downloads_folder, filename) + filepath = os.path.abspath(filepath) + try: + # 使用requests下载文件 + response = requests.get(url, stream=True) + response.raise_for_status() + + with open(filepath, 'wb') as f: + for chunk in response.iter_content(chunk_size=8192): + if chunk: + f.write(chunk) + + # 显示下载成功页面 + message = f""" +

Successfully downloaded '{filename}' to local path: +

{_local_to_docker(filepath)}

""" + + goto( + "data:text/html;base64," + + base64.b64encode(message.encode("utf-8")).decode("utf-8") + ) + + # 触发pageshow事件 + page.evaluate(""" + const event = new Event('pageshow', { + bubbles: true, + cancelable: false + }); + window.dispatchEvent(event); + """) + + except Exception as e: + raise Exception(f"Download error: {str(e)}") + else: + raise e_outer + +# def _click_id(bid: str, button: Literal["left", "middle", "right"] = "left"): +# """ +# Clicks the mouse on the target with the given element bid. + +# Examples: +# _click_id('12') +# _click_id('12', button='left') +# """ +# from typing import Dict, Union, cast +# try: +# elem = get_elem_by_bid(page, bid, demo_mode != "off") +# box = cast(Dict[str, Union[int, float]], elem.bounding_box()) +# # 如果既不是下载也不是新页面，在当前页面处理 +# page.mouse.click(box["x"] + box["width"] / 2, box["y"] + box["height"] / 2, button=button) +# try: +# page.wait_for_load_state("networkidle", timeout=5000) +# except: +# pass +# return + +# except Exception as e: +# raise Exception(f"Click error: {str(e)}") + + +def _click_id(bid: str, button: Literal["left", "middle", "right"] = "left"): + """ + Clicks the mouse on the target with the given element bid. + + Examples: + _click_id('12') + _click_id('12', button='left') + """ + # def _local_to_docker(local_path: str): + # """ + # Convert a local path to a docker path + # local_path: the local path to convert, like `{LOCAL_ROOT}/{DOCKER_WORKPLACE_NAME}/downloads/xxx` + # docker_path: the docker path to convert, like `/{DOCKER_WORKPLACE_NAME}/downloads/xxx` + # """ + # assert LOCAL_ROOT in local_path, f"local_path must contain {LOCAL_ROOT}" + # return local_path.replace(LOCAL_ROOT, '') + from typing import Dict, Union, cast + import time + import base64 + import os + from playwright._impl._api_types import TimeoutError as playwright_TimeoutError + try: + global page + elem = get_elem_by_bid(page, bid, demo_mode != "off") + box = cast(Dict[str, Union[int, float]], elem.bounding_box()) + + # 获取当前页面URL + current_url = page.url + page.context.add_cookies(COOKIES_LIST) + # goto(url) + page.set_extra_http_headers({ + "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36", + "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8", + "Accept-Language": "en-US,en;q=0.9" + }) + + # 执行点击并等待下载 + try: + with page.expect_download(timeout=5000) as download_info: # 增加到30秒 + page.mouse.click(box["x"] + box["width"] / 2, box["y"] + box["height"] / 2, button=button) + download = download_info.value + print(f"Downloading file: {download.suggested_filename}") + # 确保下载目录存在 + + download_path = f"{local_workplace}/downloads" + os.makedirs(download_path, exist_ok=True) + # 保存文件 + filepath = os.path.join(download_path, download.suggested_filename) + filepath = os.path.abspath(filepath) + download.save_as(filepath) + # 显示下载成功页面 + + message = f""" +

Successfully downloaded '{download.suggested_filename}' to local path: +

{_local_to_docker(filepath)}

""" + + goto( + "data:text/html;base64," + + base64.b64encode(message.encode("utf-8")).decode("utf-8") + ) + + # 触发pageshow事件 + page.evaluate(""" + const event = new Event('pageshow', { + bubbles: true, + cancelable: false + }); + window.dispatchEvent(event); + """) + return + except playwright_TimeoutError: + # print("Download timeout, trying alternative approach...") + # # 如果超时，尝试获取PDF直接URL并下载 + # if "arxiv.org" in current_url: + # paper_id = current_url.split("/")[-1] + # pdf_url = f"https://arxiv.org/pdf/{paper_id}.pdf" + # _visit_page(pdf_url) + # return + pass + + # 等待可能的新标签页或导航 + time.sleep(1) + + # 检查是否有新标签页 + pages_after = len(page.context.pages) + if pages_after > 1: + # 切换到最新的标签页 + page = page.context.pages[-1] + page.bring_to_front() + elif page.url != current_url: + # URL改变了，说明发生了导航 + try: + page.wait_for_load_state("networkidle", timeout=5000) + if page.get_by_text("Verify you are human by completing the action below.").count() > 0: + _checkMeetChallenge() + # 等待页面完全加载 + # 增加等待时间，确保页面完全加载 + page.wait_for_load_state("networkidle", timeout=3000) + except: + pass + + return + + except Exception as e: + raise Exception(f"Click error: {str(e)}, {type(e)}") +def _checkMeetChallenge(): + """ + check if meet challenge + + Examples: + _checkMeetChallenge() + """ + global page + def tryToClickChallenge(this_page): + try: + # 尝试定位并点击验证框架中的复选框 + frame = this_page.frame_locator("iframe[title*='challenge']") + if frame: + checkbox = frame.locator("input[type='checkbox']") + if checkbox.is_visible(): + checkbox.click() + return True + + # 尝试点击验证按钮 (同时支持中英文) + verify_texts = ["请完成以下操作，验证您是真人。", "Verify you are human by completing the action below."] + for text in verify_texts: + verify_button = this_page.get_by_text(text) + if verify_button.is_visible(): + verify_button.click() + return True + + # 尝试点击任何可见的验证按钮 + challenge_buttons = this_page.locator("button[class*='challenge']") + if challenge_buttons.count() > 0: + challenge_buttons.first.click() + return True + + except Exception as e: + print(f"尝试点击验证失败: {str(e)}") + return False + + check_count = 1 + max_attempts = 6 + while check_count <= max_attempts: + # 检查是否存在验证页面的特征 (同时支持中英文) + if (page.get_by_text("请完成以下操作，验证您是真人。").count() == 0 and + page.get_by_text("Verify you are human by completing the action below.").count() == 0): + print("验证已完成") + break + + print(f"检测到 Cloudflare 验证页面，尝试处理... (第 {check_count}/{max_attempts} 次)") + + # 尝试处理验证 + if tryToClickChallenge(page): + print("已尝试点击验证按钮，等待响应...") + + # 等待验证结果 + try: + # 等待验证页面消失或出现新内容 + page.wait_for_function(""" + () => !document.querySelector("div#challenge-stage") || + (!document.body.textContent.includes("请完成以下操作，验证您是真人。") && + !document.body.textContent.includes("Verify you are human by completing the action below.")) + """, timeout=20000) + except: + print("等待验证超时") + + # 检查是否仍在验证页面 + if check_count >= max_attempts: + if (page.get_by_text("请完成以下操作，验证您是真人。").count() > 0 or + page.get_by_text("Verify you are human by completing the action below.").count() > 0): + raise Exception("cannot pass challenge, need to restart") + + check_count += 1 + page.wait_for_timeout(5000) # 短暂等待后再次检查 + +class BrowserEnv: + def __init__(self, browsergym_eval_env: str | None = None, local_root: str | None = None, workplace_name: str | None = None): + timestamp = datetime.now().strftime("%Y%m%d_%H%M%S") + log_dir = Path(f"logs/res_{timestamp}") + log_dir.mkdir(parents=True, exist_ok=True) # recursively create all necessary parent directories + log_path = str(log_dir / "browser_env.log") + self.log_path = log_path + # self.logger = LoggerManager.get_logger() + self.html_text_converter = self.get_html_text_converter() + self.eval_mode = False + self.eval_dir = '' + self.local_workplace = os.path.join(local_root, workplace_name) + self.docker_workplace = f"/{workplace_name}" + + # EVAL only: browsergym_eval_env must be provided for evaluation + self.browsergym_eval_env = browsergym_eval_env + self.eval_mode = bool(browsergym_eval_env) + + # Initialize browser environment process + multiprocessing.set_start_method('spawn', force=True) + self.browser_side, self.agent_side = multiprocessing.Pipe() + + # tmp_env = gym.make(self.browsergym_eval_env,tags_to_mark='all') if self.eval_mode else gym.make('browsergym/openended',task_kwargs={'start_url': 'about:blank', 'goal': 'PLACEHOLDER_GOAL'}, + # wait_for_user_message=False, + # headless=True, + # disable_env_checker=True, + # tags_to_mark='all' + # ) + # obs, info = tmp_env.reset() + # self.viewport = tmp_env.env.viewport if tmp_env.env.viewport else tmp_env.env.task.viewport + # tmp_env.close() + self.init_browser() + + atexit.register(self.close) + + def get_html_text_converter(self): + html_text_converter = html2text.HTML2Text() + # ignore links and images + html_text_converter.ignore_links = False + html_text_converter.ignore_images = True + # use alt text for images + html_text_converter.images_to_alt = True + # disable auto text wrapping + html_text_converter.body_width = 0 + return html_text_converter + + @tenacity.retry( + wait=tenacity.wait_fixed(1), + stop=tenacity.stop_after_attempt(5) | stop_if_should_exit(), + retry=tenacity.retry_if_exception_type(BrowserInitException), + ) + def init_browser(self): + debug_print(True, "Starting browser env...", title = "Browser Env", log_path=self.log_path) + # self.logger.info("Starting browser env...", title="Browser Env", color="green") + try: + self.process = multiprocessing.Process(target=self.browser_process) + self.process.start() + except Exception as e: + debug_print(True, f'Failed to start browser process: {e}', title = "Browser Env", log_path=self.log_path) + # self.logger.info(f'Failed to start browser process: {e}', title="Browser Env", color="red") + raise + + if not self.check_alive(): + self.close() + raise BrowserInitException('Failed to start browser environment.') + + def browser_process(self): + if self.eval_mode: + assert self.browsergym_eval_env is not None + debug_print(True, 'Initializing browser env for web browsing evaluation.', title = "Browser Env", log_path=self.log_path) + # self.logger.info('Initializing browser env for web browsing evaluation.', title="Browser Env", color="green") + if 'webarena' in self.browsergym_eval_env: + import browsergym.webarena # noqa F401 register webarena tasks as gym environments + elif 'miniwob' in self.browsergym_eval_env: + import browsergym.miniwob # noqa F401 register miniwob tasks as gym environments + else: + raise ValueError( + f'Unsupported browsergym eval env: {self.browsergym_eval_env}' + ) + env = gym.make( + self.browsergym_eval_env, + tags_to_mark='all', + ) + else: + from browsergym.core.action.highlevel import HighLevelActionSet + def _local_to_docker(local_path: str): + """ + Convert a local path to a docker path + local_path: the local path to convert, like `{local_workplace}/downloads/xxx` + docker_path: the docker path to convert, like `{docker_workplace}/downloads/xxx` + + Examples: + _local_to_docker('/Users/tangjiabin/Documents/reasoning/autoagent/workplace_gaia_eval/downloads/xxx') + """ + local_workplace = None + docker_workplace = None + assert local_workplace in local_path, f"local_path must contain {local_workplace}" + return local_path.replace(local_workplace, docker_workplace) + source = inspect.getsource(_local_to_docker) + normalized_source = textwrap.dedent(source) + normalized_source = normalized_source.replace('local_workplace = None', f'local_workplace = {repr(self.local_workplace)}') + normalized_source = normalized_source.replace('docker_workplace = None', f'docker_workplace = {repr(self.docker_workplace)}') + + action_set = HighLevelActionSet(subsets = ["chat", "infeas", "bid", "nav", "tab", "custom"], custom_actions = [_visit_page, _click_id, _get_page_markdown, _checkMeetChallenge]) +# action_set.python_includes = \ +# f""" +# {repr(read_file('autoagent/environment/markdown_browser/mdconvert.py'))} + +# """ + action_set.python_includes + + action_set.python_includes = f"""\ +{convert_cookies_to_python()} +""" + action_set.python_includes + action_set.python_includes = f"""\ +def _local_to_docker(local_path: str): + local_workplace = {repr(self.local_workplace)} + docker_workplace = {repr(self.docker_workplace)} + assert local_workplace in local_path + return local_path.replace(local_workplace, docker_workplace) + +""" + action_set.python_includes + action_set.python_includes = f"local_workplace = {repr(self.local_workplace)}\n" + action_set.python_includes + + # action_set.python_includes = f"LOCAL_ROOT = {repr(LOCAL_ROOT)}\n" + action_set.python_includes + + # print(action_set.python_includes) + action_mapping = action_set.to_python_code + env = gym.make( + 'browsergym/openended', + task_kwargs={'start_url': 'about:blank', 'goal': 'PLACEHOLDER_GOAL'}, + wait_for_user_message=False, + headless=True, + disable_env_checker=True, + tags_to_mark='all', + action_mapping = action_mapping + ) + + + obs, info = env.reset() + + # self.viewport = env.env.viewport if env.env.viewport else env.env.task.viewport + # print(f"Viewport: {self.viewport}") + # 通过管道发送viewport信息 + + # EVAL ONLY: save the goal into file for evaluation + self.eval_goal = None + self.eval_rewards: list[float] = [] + if self.eval_mode: + debug_print(True, f"Browsing goal: {obs['goal']}", title = "Browser Env", log_path=self.log_path) + # self.logger.info(f"Browsing goal: {obs['goal']}", title="Browser Env", color="green") + self.eval_goal = obs['goal'] + + debug_print(True, 'Browser env started.', title = "Browser Env", log_path=self.log_path) + # self.logger.info('Browser env started.', title="Browser Env", color="green") + while should_continue(): + try: + if self.browser_side.poll(timeout=0.01): + unique_request_id, action_data = self.browser_side.recv() + + # shutdown the browser environment + if unique_request_id == 'SHUTDOWN': + debug_print(False, 'SHUTDOWN recv, shutting down browser env...', title = "Browser Env", log_path=self.log_path) + # self.logger.info('SHUTDOWN recv, shutting down browser env...', title="Browser Env", color="green") + env.close() + return + elif unique_request_id == 'IS_ALIVE': + self.browser_side.send(('ALIVE', None)) + continue + + # EVAL ONLY: Get evaluation info + if action_data['action'] == BROWSER_EVAL_GET_GOAL_ACTION: + self.browser_side.send( + (unique_request_id, {'text_content': self.eval_goal}) + ) + continue + elif action_data['action'] == BROWSER_EVAL_GET_REWARDS_ACTION: + self.browser_side.send( + ( + unique_request_id, + {'text_content': json.dumps(self.eval_rewards)}, + ) + ) + continue + + action = action_data['action'] + obs, reward, terminated, truncated, info = env.step(action) + + # EVAL ONLY: Save the rewards into file for evaluation + if self.eval_mode: + self.eval_rewards.append(reward) + + # add text content of the page + html_str = flatten_dom_to_str(obs['dom_object']) + obs['text_content'] = self.html_text_converter.handle(html_str) + # make observation serializable + obs['screenshot'] = self.image_to_png_base64_url(obs['screenshot']) + obs['active_page_index'] = obs['active_page_index'].item() + obs['elapsed_time'] = obs['elapsed_time'].item() + self.browser_side.send((unique_request_id, obs)) + except KeyboardInterrupt: + debug_print(True, 'Browser env process interrupted by user.', title = "Browser Env", log_path=self.log_path) + # self.logger.info('Browser env process interrupted by user.', title="Browser Env", color="green") + try: + env.close() + except Exception: + pass + return + + def step(self, action_str: str, timeout: float = 30) -> dict: + """Execute an action in the browser environment and return the observation.""" + unique_request_id = str(uuid.uuid4()) + self.agent_side.send((unique_request_id, {'action': action_str})) + start_time = time.time() + while True: + if should_exit() or (time.time() - start_time > timeout and '_visit_page' not in action_str): + raise TimeoutError('Browser environment took too long to respond.') + if should_exit() or (time.time() - start_time > 600 and '_visit_page' in action_str): + raise TimeoutError('Browser environment took too long to respond.') + if self.agent_side.poll(timeout=0.01): + response_id, obs = self.agent_side.recv() + if response_id == unique_request_id: + return obs + + def check_alive(self, timeout: float = 60): + self.agent_side.send(('IS_ALIVE', None)) + if self.agent_side.poll(timeout=timeout): + response_id, _ = self.agent_side.recv() + if response_id == 'ALIVE': + return True + debug_print(True, f'Browser env is not alive. Response ID: {response_id}', title = "Browser Env", log_path=self.log_path) + # self.logger.info(f'Browser env is not alive. Response ID: {response_id}', title="Browser Env", color="red") + + def close(self): + if not self.process.is_alive(): + return + try: + self.agent_side.send(('SHUTDOWN', None)) + self.process.join(5) # Wait for the process to terminate + if self.process.is_alive(): + debug_print(True, 'Browser process did not terminate, forcefully terminating...', title = "Browser Env", log_path=self.log_path) + # self.logger.info('Browser process did not terminate, forcefully terminating...', title="Browser Env", color="red") + self.process.terminate() + self.process.join(5) # Wait for the process to terminate + if self.process.is_alive(): + self.process.kill() + self.process.join(5) # Wait for the process to terminate + self.agent_side.close() + self.browser_side.close() + except Exception: + debug_print(True, 'Encountered an error when closing browser env', exc_info=True, title = "Browser Env", log_path=self.log_path) + # self.logger.info('Encountered an error when closing browser env', exc_info=True, title="Browser Env", color="red") + + @staticmethod + def image_to_png_base64_url( + image: np.ndarray | Image.Image, add_data_prefix: bool = False + ): + """Convert a numpy array to a base64 encoded png image url.""" + if isinstance(image, np.ndarray): + image = Image.fromarray(image) + if image.mode in ('RGBA', 'LA'): + image = image.convert('RGB') + buffered = io.BytesIO() + image.save(buffered, format='PNG') + + image_base64 = base64.b64encode(buffered.getvalue()).decode() + return ( + f'data:image/png;base64,{image_base64}' + if add_data_prefix + else f'{image_base64}' + ) + + @staticmethod + def image_to_jpg_base64_url( + image: np.ndarray | Image.Image, add_data_prefix: bool = False + ): + """Convert a numpy array to a base64 encoded jpeg image url.""" + if isinstance(image, np.ndarray): + image = Image.fromarray(image) + if image.mode in ('RGBA', 'LA'): + image = image.convert('RGB') + buffered = io.BytesIO() + image.save(buffered, format='JPEG') + + image_base64 = base64.b64encode(buffered.getvalue()).decode() + return ( + f'data:image/jpeg;base64,{image_base64}' + if add_data_prefix + else f'{image_base64}' + ) +def source_to_function(source_code: str, func_name: str): + """将源代码字符串转换为函数，支持 inspect.getsource""" + # 创建临时文件 + with tempfile.NamedTemporaryFile(mode='w', suffix='.py', delete=False) as f: + f.write(source_code) + temp_path = f.name + + try: + # 导入临时模块 + import importlib.util + spec = importlib.util.spec_from_file_location("temp_module", temp_path) + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + + # 获取函数 + func = getattr(module, func_name) + return func + + finally: + # 清理临时文件 + os.unlink(temp_path) + + + \ No newline at end of file diff --git a/autoagent/environment/cookie_json/README.md b/autoagent/environment/cookie_json/README.md new file mode 100644 index 0000000..a600a95 --- /dev/null +++ b/autoagent/environment/cookie_json/README.md @@ -0,0 +1,36 @@ +# How to obtain cookie json files + +## What are cookies? +Cookies are small pieces of data stored by websites on users' computers, containing information like login status and preferences. They are essential for web automation as they allow automated browsers to maintain authenticated sessions, skip repeated logins, and simulate real user behavior across multiple page visits. + +## How to organize them in this folder? +We recommend you to use the Google Chrome browser with the extension "Export cookie JSON file for Puppeteer", as show in the following figure: + +![extension](../../../assets/cookies/extension.png) + +1. Go to a specific website and login. +2. Then use the extension to export the cookies, and save it as a json file in the `cookie_json` folder. + +![export](../../../assets/cookies/export.png) + +3. After you have exported all cookies, use the following command to convert them to python code: + +```bash +cd path/to/MetaChain && python autoagent/environment/browser_cookies.py +``` + +## Recommended websites + +We recommend you to export the cookies from the following websites: + +- [archive.org](https://archive.org) +- [github.com](https://github.com) +- [nature.com](https://nature.com) +- [orcid.org](https://orcid.org) +- [www.collinsdictionary.com](https://www.collinsdictionary.com) +- [www.jstor.org](https://www.jstor.org) +- [www.ncbi.nlm.nih.gov](https://www.ncbi.nlm.nih.gov) +- [www.pnas.org](https://www.pnas.org) +- [www.reddit.com](https://www.reddit.com) +- [www.researchgate.net](https://www.researchgate.net) +- [www.youtube.com](https://www.youtube.com) diff --git a/autoagent/environment/cookies_data.py b/autoagent/environment/cookies_data.py new file mode 100644 index 0000000..e1453a6 --- /dev/null +++ b/autoagent/environment/cookies_data.py @@ -0,0 +1,2 @@ +COOKIES_LIST = [ +] diff --git a/autoagent/environment/docker_env.py b/autoagent/environment/docker_env.py new file mode 100644 index 0000000..12f0d5d --- /dev/null +++ b/autoagent/environment/docker_env.py @@ -0,0 +1,276 @@ +import os +import os.path as osp +import subprocess +from constant import BASE_IMAGES, AI_USER, GITHUB_AI_TOKEN +import time +import socket +import json +from pathlib import Path +import shutil +wd = Path(__file__).parent.resolve() +from dataclasses import dataclass, field +from typing import Optional, Union, Dict +from functools import update_wrapper +from inspect import signature +@dataclass +class DockerConfig: + container_name: str + workplace_name: str + communication_port: int # 12345 + conda_path: str # /root/miniconda3 + test_pull_name: str = field(default='main') + task_name: Optional[str] = field(default=None) + git_clone: bool = field(default=False) + setup_package: Optional[str] = field(default=None) + local_root: str = field(default=os.getcwd()) + + +class DockerEnv: + def __init__(self, config: Union[DockerConfig, Dict]): + if isinstance(config, Dict): + config = DockerConfig(**config) + self.workplace_name = config.workplace_name + self.local_workplace = osp.join(config.local_root, config.workplace_name) + self.docker_workplace = f"/{config.workplace_name}" + self.container_name = config.container_name + self.test_pull_name = config.test_pull_name + self.task_name = config.task_name + self.git_clone = config.git_clone + self.setup_package = config.setup_package + self.communication_port = config.communication_port + self.conda_path = config.conda_path + + def init_container(self): + container_check_command = ["docker", "ps", "-a", "--filter", f"name={self.container_name}", "--format", "{{.Names}}"] + existing_container = subprocess.run(container_check_command, capture_output=True, text=True) + os.makedirs(self.local_workplace, exist_ok=True) + + if not osp.exists(osp.join(self.local_workplace, 'tcp_server.py')): + shutil.copy(osp.join(wd, 'tcp_server.py'), self.local_workplace) + assert osp.exists(osp.join(self.local_workplace, 'tcp_server.py')), "Failed to copy tcp_server.py to the local workplace" + if self.setup_package is not None: + unzip_command = ["tar", "-xzvf", f"packages/{self.setup_package}.tar.gz", "-C", self.local_workplace] + subprocess.run(unzip_command) + if self.git_clone: + if not os.path.exists(os.path.join(self.local_workplace, 'AutoAgent')): + git_command = ["cd", self.local_workplace, "&&", "git", "clone", "-b", self.test_pull_name, f"https://{AI_USER}:{GITHUB_AI_TOKEN}@github.com/HKUDS/AutoAgent.git"] + print(git_command) + git_command = " ".join(git_command) + + result = subprocess.run(git_command, shell=True) + if result.returncode != 0: + raise Exception(f"Failed to clone the repository. The error is: {result.stdout}") + copy_env_command = f"cp .env {self.local_workplace}/AutoAgent" + result = subprocess.run(copy_env_command, shell=True, capture_output=True, text=True) + if result.returncode != 0: + raise Exception(f"Failed to copy .env file to the AutoAgent directory. Error: {result.stderr}") + # create a new branch + new_branch_name = f"{self.test_pull_name}_{self.task_name}" + create_branch_command = f"cd {self.local_workplace}/AutoAgent && git checkout -b {new_branch_name}" + result = subprocess.run(create_branch_command, shell=True, capture_output=True, text=True) + if result.returncode != 0: + print(Exception(f"Failed to create and switch to new branch. Error: {result.stderr}")) + switch_branch_command = f"cd {self.local_workplace}/AutoAgent && git checkout {new_branch_name}" + result = subprocess.run(switch_branch_command, shell=True, capture_output=True, text=True) + if result.returncode != 0: + raise Exception(f"Failed to switch to new branch. Error: {result.stderr}") + else: + print(f"Successfully switched to new branch: {new_branch_name}") + else: + print(f"Successfully created and switched to new branch: {new_branch_name}") + + if existing_container.stdout.strip() == self.container_name: + # check if the container is running + running_check_command = ["docker", "ps", "--filter", f"name={self.container_name}", "--format", "{{.Names}}"] + running_container = subprocess.run(running_check_command, capture_output=True, text=True) + + if running_container.stdout.strip() == self.container_name: + print(f"Container '{self.container_name}' is already running. Skipping creation.") + return # container is already running, skip creation + else: + # container exists but is not running, start it + start_command = ["docker", "start", self.container_name] + subprocess.run(start_command) + print(f"Container '{self.container_name}' has been started.") + return + + # if the container does not exist, create and start a new container + docker_command = [ + "docker", "run", "-d", "--name", self.container_name, "--user", "root", + "-v", f"{self.local_workplace}:{self.docker_workplace}", + "-w", f"{self.docker_workplace}", "-p", f"{self.communication_port}:{self.communication_port}", BASE_IMAGES, + "/bin/bash", "-c", + f"python3 {self.docker_workplace}/tcp_server.py --workplace {self.workplace_name} --conda_path {self.conda_path} --port {self.communication_port}" + ] + # execute the docker command + result = subprocess.run(docker_command, capture_output=True, text=True) + if result.returncode != 0: + raise Exception(f"Failed to start container: {result.stderr}") + if self.wait_for_container_ready(timeout=60): + print(f"Container '{self.container_name}' has been created and started.") + def wait_for_container_ready(self, timeout=30): + """using subprocess to check if the container is running""" + start_time = time.time() + + while time.time() - start_time < timeout: + result = subprocess.run( + ["docker", "inspect", "--format", "{{.State.Running}}", self.container_name], + capture_output=True, + text=True + ) + + if result.returncode == 0 and "true" in result.stdout.lower(): + # 额外检查 tcp_server 是否运行 + try: + port_info = check_container_ports(self.container_name) + assert port_info and (port_info[0] == port_info[1]) + available_port = port_info[0] + self.communication_port = available_port + result = self.run_command('ps aux') + if "tcp_server.py" in result['result']: + return True + except Exception as e: + pass + + time.sleep(1) + + raise TimeoutError(f"Container {self.container_name} failed to start within {timeout} seconds") + def stop_container(self): + stop_command = ["docker", "stop", self.container_name] + result = subprocess.run(stop_command, capture_output=True, text=True) + if result.returncode != 0: + raise Exception(f"Failed to stop container: {result.stderr}") + + def run_command(self, command, stream_callback=None): + """ + communicate with docker container and execute command, support stream output + + Args: + command: the command to execute + stream_callback: optional callback function, for handling stream output + the function signature should be callback(text: str) + + Returns: + dict: the complete JSON result returned by the docker container + """ + hostname = 'localhost' + port = self.communication_port + buffer_size = 4096 + + with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s: + s.connect((hostname, port)) + s.sendall(command.encode()) + + partial_line = "" + while True: + chunk = s.recv(buffer_size) + if not chunk: + break + + # add new received data to the unfinished data + data = partial_line + chunk.decode('utf-8') + lines = data.split('\n') + + # except the last line, process all complete lines + for line in lines[:-1]: + if line: + try: + response = json.loads(line) + if response['type'] == 'chunk': + # process stream output + if stream_callback: + stream_callback(response['data']) + elif response['type'] == 'final': + # return the final result + return { + 'status': response['status'], + 'result': response['result'] + } + except json.JSONDecodeError: + print(f"Invalid JSON: {line}") + + # save the possibly unfinished last line + partial_line = lines[-1] + + # if the loop ends normally without receiving a final response + return { + 'status': -1, + 'result': 'Connection closed without final response' + } + +def with_env(env: DockerEnv): + """将env注入到工具函数中的装饰器""" + def decorator(func): + def wrapped(*args, **kwargs): + return func(env=env, *args, **kwargs) + + # 保留原始函数的所有属性 + update_wrapper(wrapped, func) + # 修改signature，移除env参数 + wrapped.__signature__ = signature(func).replace( + parameters=[p for p in signature(func).parameters.values() if p.name != 'env'] + ) + if func.__doc__: + try: + if '{docker_workplace}' in func.__doc__: + wrapped.__doc__ = func.__doc__.format(docker_workplace=env.docker_workplace) + else: + wrapped.__doc__ = func.__doc__ + if '{local_workplace}' in func.__doc__: + wrapped.__doc__ = func.__doc__.format(local_workplace=env.local_workplace) + else: + wrapped.__doc__ = func.__doc__ + except (KeyError, IndexError, ValueError): + # 如果格式化失败（没有占位符），保持原始文档 + wrapped.__doc__ = func.__doc__ + return wrapped + return decorator + +def check_container_ports(container_name: str): + """ + check if the container has port mapping + return format: + - if the container exists and has port mapping: '0.0.0.0:12345->12345/tcp' + - if the container does not exist or does not have port mapping: None + """ + # use docker ps to check the container and get the port information + container_check_command = [ + "docker", "ps", "-a", + "--filter", f"name={container_name}", + "--format", "{{.Ports}}" + ] + + result = subprocess.run(container_check_command, capture_output=True, text=True) + ports_info = result.stdout.strip() + + if not ports_info: + return None + + # only process the mapped ports + for mapping in ports_info.split(','): + mapping = mapping.strip() + if '->' in mapping: + # parse '0.0.0.0:12345->12345/tcp' to (12345, 12345) + host_part, container_part = mapping.split('->') + host_port = host_part.split(':')[1] # get '12345' from '0.0.0.0:12345' + container_port = container_part.split('/')[0] # get '12345' from '12345/tcp' + return (int(host_port), int(container_port)) # convert to integers + return None + +def check_container_exist(container_name: str): + container_check_command = [ + "docker", "ps", "-a", + "--filter", f"name={container_name}", + "--format", "{{.Names}}" + ] + result = subprocess.run(container_check_command, capture_output=True, text=True) + return container_name in result.stdout.strip() + +def check_container_running(container_name: str): + container_check_command = [ + "docker", "ps", + "--filter", f"name={container_name}", + "--format", "{{.Names}}" + ] + result = subprocess.run(container_check_command, capture_output=True, text=True) + return container_name in result.stdout.strip() diff --git a/autoagent/environment/local_env.py b/autoagent/environment/local_env.py new file mode 100644 index 0000000..5143b54 --- /dev/null +++ b/autoagent/environment/local_env.py @@ -0,0 +1,101 @@ +import subprocess +import json +import os +from pathlib import Path +import platform +import os.path as osp +from autoagent.environment.docker_env import DockerConfig +class LocalEnv: + def __init__(self, docker_config: DockerConfig = None): + if docker_config is None: + self.docker_workplace = os.getcwd() + if self.docker_workplace.endswith("autoagent"): + self.docker_workplace = os.path.dirname(self.docker_workplace) + self.local_workplace = self.docker_workplace + else: + self.local_workplace = osp.join(docker_config.local_root, docker_config.workplace_name) + self.docker_workplace = osp.join(docker_config.local_root, docker_config.workplace_name) + os.makedirs(self.local_workplace, exist_ok=True) + self.conda_sh = self._find_conda_sh() + + + def _find_conda_sh(self) -> str: + """ + Find conda.sh file location across different environments + """ + + # 1. Try common locations based on OS + possible_paths = [] + home = str(Path.home()) + + if platform.system() == "Windows": + possible_paths.extend([ + Path(home) / "Anaconda3" / "etc" / "profile.d" / "conda.sh", + Path(home) / "miniconda3" / "etc" / "profile.d" / "conda.sh", + Path(home) / "micromamba" / "etc" / "profile.d" / "conda.sh", + ]) + else: # Linux and MacOS + possible_paths.extend([ + Path(home) / "anaconda3" / "etc" / "profile.d" / "conda.sh", + Path(home) / "miniconda3" / "etc" / "profile.d" / "conda.sh", + Path(home) / "micromamba" / "etc" / "profile.d" / "conda.sh", + Path("/opt/conda/etc/profile.d/conda.sh"), # Docker containers + Path("/usr/local/conda/etc/profile.d/conda.sh"), + ]) + + # For Linux, also check root installations + if platform.system() == "Linux": + possible_paths.extend([ + Path("/opt/anaconda3/etc/profile.d/conda.sh"), + Path("/opt/miniconda3/etc/profile.d/conda.sh"), + Path("/opt/micromamba/etc/profile.d/conda.sh"), + ]) + + # Check all possible paths + for path in possible_paths: + if path.exists(): + return str(path) + + # 2. Try to find using conda info command + try: + result = subprocess.run(['conda', 'info', '--base'], + capture_output=True, + text=True) + if result.returncode == 0: + base_path = result.stdout.strip() + conda_sh = Path(base_path) / "etc" / "profile.d" / "conda.sh" + if conda_sh.exists(): + return str(conda_sh) + except: + pass + + # 3. If all fails, return None and handle in run_command + return None + def run_command(self, command, stream_callback=None): + assert self.conda_sh is not None, "Conda.sh not found" + modified_command = f"/bin/bash -c 'source {self.conda_sh} && conda activate auto && cd {self.docker_workplace} && {command}'" + process = subprocess.Popen(modified_command, shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True) + output = '' + while True: + line = process.stdout.readline() + if not line and process.poll() is not None: + break + output += line + # 立即发送每一行输出 + + # 发送最终的完整响应 + response = { + "status": process.poll(), + "result": output + } + return response + def _convert_local_to_docker(self, path): + return path + + def _convert_docker_to_local(self, path): + return path + +if __name__ == "__main__": + # print(str(Path.home())) + local_env = LocalEnv() + print(local_env.conda_sh) diff --git a/autoagent/environment/markdown_browser/__init__.py b/autoagent/environment/markdown_browser/__init__.py new file mode 100644 index 0000000..5b6880e --- /dev/null +++ b/autoagent/environment/markdown_browser/__init__.py @@ -0,0 +1,22 @@ +from .abstract_markdown_browser import AbstractMarkdownBrowser +from .markdown_search import AbstractMarkdownSearch, BingMarkdownSearch + +# TODO: Fix mdconvert +from .mdconvert import ( # type: ignore + DocumentConverterResult, + FileConversionException, + MarkdownConverter, + UnsupportedFormatException, +) +from .requests_markdown_browser import RequestsMarkdownBrowser + +__all__ = ( + "AbstractMarkdownBrowser", + "RequestsMarkdownBrowser", + "AbstractMarkdownSearch", + "BingMarkdownSearch", + "MarkdownConverter", + "UnsupportedFormatException", + "FileConversionException", + "DocumentConverterResult", +) diff --git a/autoagent/environment/markdown_browser/abstract_markdown_browser.py b/autoagent/environment/markdown_browser/abstract_markdown_browser.py new file mode 100644 index 0000000..70417a6 --- /dev/null +++ b/autoagent/environment/markdown_browser/abstract_markdown_browser.py @@ -0,0 +1,64 @@ +from abc import ABC, abstractmethod +from typing import Union + + +class AbstractMarkdownBrowser(ABC): + """ + An abstract class for a Markdown web browser. + + All MarkdownBrowers work by: + + (1) fetching a web page by URL (via requests, Selenium, Playwright, etc.) + (2) converting the page's HTML or DOM to Markdown + (3) operating on the Markdown + + Such browsers are simple, and suitable for read-only agentic use. + They cannot be used to interact with complex web applications. + """ + + @abstractmethod + def __init__(self) -> None: + pass + + @property + @abstractmethod + def address(self) -> str: + pass + + @abstractmethod + def set_address(self, uri_or_path: str) -> None: + pass + + @property + @abstractmethod + def viewport(self) -> str: + pass + + @property + @abstractmethod + def page_content(self) -> str: + pass + + @abstractmethod + def page_down(self) -> None: + pass + + @abstractmethod + def page_up(self) -> None: + pass + + @abstractmethod + def visit_page(self, path_or_uri: str) -> str: + pass + + @abstractmethod + def open_local_file(self, local_path: str) -> str: + pass + + @abstractmethod + def find_on_page(self, query: str) -> Union[str, None]: + pass + + @abstractmethod + def find_next(self) -> Union[str, None]: + pass diff --git a/autoagent/environment/markdown_browser/markdown_search.py b/autoagent/environment/markdown_browser/markdown_search.py new file mode 100644 index 0000000..1ce56f5 --- /dev/null +++ b/autoagent/environment/markdown_browser/markdown_search.py @@ -0,0 +1,282 @@ +import logging +import os +import re +from abc import ABC, abstractmethod +from typing import Any, Dict, List, Optional, cast +from urllib.parse import quote, quote_plus, unquote, urlparse, urlunparse + +import requests + +# TODO: Fix these types +from .mdconvert import MarkdownConverter # type: ignore + +logger = logging.getLogger(__name__) + + +class AbstractMarkdownSearch(ABC): + """ + An abstract class for providing search capabilities to a Markdown browser. + """ + + @abstractmethod + def search(self, query: str) -> str: + pass + + +class BingMarkdownSearch(AbstractMarkdownSearch): + """ + Provides Bing web search capabilities to Markdown browsers. + """ + + def __init__(self, bing_api_key: Optional[str] = None, interleave_results: bool = True): + """ + Perform a Bing web search, and return the results formatted in Markdown. + + Args: + bing_api_key: key for the Bing search API. If omitted, an attempt is made to read the key from the BING_API_KEY environment variable. If no key is found, BingMarkdownSearch will print a warning, and will fall back to visiting and scraping the live Bing results page. Scraping is objectively worse than using the API, and thus is not recommended. + interleave_results: When using the Bing API, results are returned based on category (web, news, videos, etc.), along with instructions for how they should be interleaved on the page. When `interleave` is set to True, these interleaving instructions are followed, and a single results list is returned by BingMarkdownSearch. When `interleave` is set to false, results are separated by category, and no interleaving is done. + """ + self._mdconvert = MarkdownConverter() + self._interleave_results = interleave_results + + if bing_api_key is None or bing_api_key.strip() == "": + self._bing_api_key = os.environ.get("BING_API_KEY") + else: + self._bing_api_key = bing_api_key + + if self._bing_api_key is None: + if not self._interleave_results: + raise ValueError( + "No Bing API key was provided. This is incompatible with setting `interleave_results` to False. Please provide a key, or set `interleave_results` to True." + ) + + # logger.warning( + # "Warning: No Bing API key provided. BingMarkdownSearch will submit an HTTP request to the Bing landing page, but results may be missing or low quality. To resolve this warning, provide a Bing API key by setting the BING_API_KEY environment variable, or using the 'bing_api_key' parameter in by BingMarkdownSearch's constructor. Bing API keys can be obtained via https://www.microsoft.com/en-us/bing/apis/bing-web-search-api\n" + # ) + + def search(self, query: str) -> str: + """Search Bing and return the results formatted in Markdown. If a Bing API key is available, the API is used to perform the search. If no API key is available, the search is performed by submitting an HTTPs GET request directly to Bing. Searches performed with the API are much higher quality, and are more reliable. + + Args: + query: The search query to issue + + Returns: + A Markdown rendering of the search results. + """ + + if self._bing_api_key is None: + return self._fallback_search(query) + else: + return self._api_search(query) + + def _api_search(self, query: str) -> str: + """Search Bing using the API, and return the results formatted in Markdown. + + Args: + query: The search query to issue + + Returns: + A Markdown rendering of the search results. + """ + results = self._bing_api_call(query) + + snippets: Dict[str, List[str]] = dict() + + def _processFacts(elm: List[Dict[str, Any]]) -> str: + facts: List[str] = list() + for e in elm: + k = e["label"]["text"] + v = " ".join(item["text"] for item in e["items"]) + facts.append(f"{k}: {v}") + return "\n".join(facts) + + # Web pages + # __POS__ is a placeholder for the final ranking position, added at the end + web_snippets: List[str] = list() + if "webPages" in results: + for page in results["webPages"]["value"]: + snippet = f"__POS__. {self._markdown_link(page['name'], page['url'])}\n{page['snippet']}" + + if "richFacts" in page: + snippet += "\n" + _processFacts(page["richFacts"]) + + if "mentions" in page: + snippet += "\nMentions: " + ", ".join(e["name"] for e in page["mentions"]) + + if page["id"] not in snippets: + snippets[page["id"]] = list() + snippets[page["id"]].append(snippet) + web_snippets.append(snippet) + + if "deepLinks" in page: + for dl in page["deepLinks"]: + deep_snippet = f"__POS__. {self._markdown_link(dl['name'], dl['url'])}\n{dl['snippet'] if 'snippet' in dl else ''}" + snippets[page["id"]].append(deep_snippet) + web_snippets.append(deep_snippet) + + # News results + news_snippets: List[str] = list() + if "news" in results: + for page in results["news"]["value"]: + snippet = ( + f"__POS__. {self._markdown_link(page['name'], page['url'])}\n{page.get('description', '')}".strip() + ) + + if "datePublished" in page: + snippet += "\nDate published: " + page["datePublished"].split("T")[0] + + if "richFacts" in page: + snippet += "\n" + _processFacts(page["richFacts"]) + + if "mentions" in page: + snippet += "\nMentions: " + ", ".join(e["name"] for e in page["mentions"]) + + news_snippets.append(snippet) + + if len(news_snippets) > 0: + snippets[results["news"]["id"]] = news_snippets + + # Videos + video_snippets: List[str] = list() + if "videos" in results: + for page in results["videos"]["value"]: + if not page["contentUrl"].startswith("https://www.youtube.com/watch?v="): + continue + + snippet = f"__POS__. {self._markdown_link(page['name'], page['contentUrl'])}\n{page.get('description', '')}".strip() + + if "datePublished" in page: + snippet += "\nDate published: " + page["datePublished"].split("T")[0] + + if "richFacts" in page: + snippet += "\n" + _processFacts(page["richFacts"]) + + if "mentions" in page: + snippet += "\nMentions: " + ", ".join(e["name"] for e in page["mentions"]) + + video_snippets.append(snippet) + + if len(video_snippets) > 0: + snippets[results["videos"]["id"]] = video_snippets + + # Related searches + related_searches = "" + if "relatedSearches" in results: + related_searches = "## Related Searches:\n" + for s in results["relatedSearches"]["value"]: + related_searches += "- " + s["text"] + "\n" + snippets[results["relatedSearches"]["id"]] = [related_searches.strip()] + + idx = 0 + content = "" + if self._interleave_results: + # Interleaved + for item in results["rankingResponse"]["mainline"]["items"]: + _id = item["value"]["id"] + if _id in snippets: + for s in snippets[_id]: + if "__POS__" in s: + idx += 1 + content += s.replace("__POS__", str(idx)) + "\n\n" + else: + content += s + "\n\n" + else: + # Categorized + if len(web_snippets) > 0: + content += "## Web Results\n\n" + for s in web_snippets: + if "__POS__" in s: + idx += 1 + content += s.replace("__POS__", str(idx)) + "\n\n" + else: + content += s + "\n\n" + if len(news_snippets) > 0: + content += "## News Results\n\n" + for s in news_snippets: + if "__POS__" in s: + idx += 1 + content += s.replace("__POS__", str(idx)) + "\n\n" + else: + content += s + "\n\n" + if len(video_snippets) > 0: + content += "## Video Results\n\n" + for s in video_snippets: + if "__POS__" in s: + idx += 1 + content += s.replace("__POS__", str(idx)) + "\n\n" + else: + content += s + "\n\n" + if len(related_searches) > 0: + content += related_searches + + return f"## A Bing search for '{query}' found {idx} results:\n\n" + content.strip() + + def _bing_api_call(self, query: str) -> Dict[str, Any]: + """Make a Bing API call, and return a Python representation of the JSON response." + + Args: + query: The search query to issue + + Returns: + A Python representation of the Bing API's JSON response (as parsed by `json.loads()`). + """ + # Make sure the key was set + if not self._bing_api_key: + raise ValueError("Missing Bing API key.") + + # Prepare the request parameters + request_kwargs: Dict[str, Any] = {} + request_kwargs["headers"] = {} + request_kwargs["headers"]["Ocp-Apim-Subscription-Key"] = self._bing_api_key + + request_kwargs["params"] = {} + request_kwargs["params"]["q"] = query + request_kwargs["params"]["textDecorations"] = False + request_kwargs["params"]["textFormat"] = "raw" + + request_kwargs["stream"] = False + + # Make the request + response = requests.get("https://api.bing.microsoft.com/v7.0/search", **request_kwargs) + response.raise_for_status() + results = response.json() + + return cast(Dict[str, Any], results) + + def _fallback_search(self, query: str) -> str: + """When no Bing API key is provided, we issue a simple HTTPs GET call to the Bing landing page and convert it to Markdown. + + Args: + query: The search query to issue + + Returns: + The Bing search results page, converted to Markdown. + """ + user_agent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/119.0.0.0 Safari/537.36 Edg/119.0.0.0" + headers = {"User-Agent": user_agent} + + url = f"https://www.bing.com/search?q={quote_plus(query)}&FORM=QBLH" + response = requests.get(url, headers=headers) + response.raise_for_status() + # TODO: Fix the types + return self._mdconvert.convert_response(response).text_content # type: ignore + + def _markdown_link(self, anchor: str, href: str) -> str: + """Create a Markdown hyperlink, escaping the URLs as appropriate. + + Args: + anchor: The anchor text of the hyperlink + href: The href destination of the hyperlink + + Returns: + A correctly-formatted Markdown hyperlink + """ + try: + parsed_url = urlparse(href) + # URLs provided by Bing are sometimes only partially quoted, leaving in characters + # the conflict with Markdown. We unquote the URL, and then re-quote more completely + href = urlunparse(parsed_url._replace(path=quote(unquote(parsed_url.path)))) + anchor = re.sub(r"[\[\]]", " ", anchor) + return f"[{anchor}]({href})" + except ValueError: # It's not clear if this ever gets thrown + return f"[{anchor}]({href})" diff --git a/autoagent/environment/markdown_browser/mdconvert.py b/autoagent/environment/markdown_browser/mdconvert.py new file mode 100644 index 0000000..196c452 --- /dev/null +++ b/autoagent/environment/markdown_browser/mdconvert.py @@ -0,0 +1,1033 @@ +# type: ignore +import base64 +import binascii +import copy +import html +import json +import mimetypes +import os +import re +import shutil +import subprocess +import sys +import tempfile +from typing import Any, Dict, List, Optional, Union +from urllib.parse import parse_qs, quote, unquote, urlparse, urlunparse + +import mammoth +import markdownify +import pandas as pd +import pdfminer +import pdfminer.high_level +import pptx +from docling.document_converter import DocumentConverter as DocLingDocumentConverter + +# File-format detection +import puremagic +import requests +from bs4 import BeautifulSoup + +# Optional Transcription support +try: + import pydub + import speech_recognition as sr + + IS_AUDIO_TRANSCRIPTION_CAPABLE = True +except ModuleNotFoundError: + pass + +# Optional YouTube transcription support +try: + from youtube_transcript_api import YouTubeTranscriptApi + + IS_YOUTUBE_TRANSCRIPT_CAPABLE = True +except ModuleNotFoundError: + pass + + +class _CustomMarkdownify(markdownify.MarkdownConverter): + """ + A custom version of markdownify's MarkdownConverter. Changes include: + + - Altering the default heading style to use '#', '##', etc. + - Removing javascript hyperlinks. + - Truncating images with large data:uri sources. + - Ensuring URIs are properly escaped, and do not conflict with Markdown syntax + """ + + def __init__(self, **options: Any): + options["heading_style"] = options.get("heading_style", markdownify.ATX) + # Explicitly cast options to the expected type if necessary + super().__init__(**options) + + def convert_hn(self, n: int, el: Any, text: str, convert_as_inline: bool) -> str: + """Same as usual, but be sure to start with a new line""" + if not convert_as_inline: + if not re.search(r"^\n", text): + return "\n" + super().convert_hn(n, el, text, convert_as_inline) # type: ignore + + return super().convert_hn(n, el, text, convert_as_inline) # type: ignore + + def convert_a(self, el: Any, text: str, convert_as_inline: bool): + """Same as usual converter, but removes Javascript links and escapes URIs.""" + prefix, suffix, text = markdownify.chomp(text) # type: ignore + if not text: + return "" + href = el.get("href") + title = el.get("title") + + # Escape URIs and skip non-http or file schemes + if href: + try: + parsed_url = urlparse(href) # type: ignore + if parsed_url.scheme and parsed_url.scheme.lower() not in ["http", "https", "file"]: # type: ignore + return "%s%s%s" % (prefix, text, suffix) + href = urlunparse(parsed_url._replace(path=quote(unquote(parsed_url.path)))) # type: ignore + except ValueError: # It's not clear if this ever gets thrown + return "%s%s%s" % (prefix, text, suffix) + + # For the replacement see #29: text nodes underscores are escaped + if ( + self.options["autolinks"] + and text.replace(r"\_", "_") == href + and not title + and not self.options["default_title"] + ): + # Shortcut syntax + return "<%s>" % href + if self.options["default_title"] and not title: + title = href + title_part = ' "%s"' % title.replace('"', r"\"") if title else "" + return "%s[%s](%s%s)%s" % (prefix, text, href, title_part, suffix) if href else text + + def convert_img(self, el: Any, text: str, convert_as_inline: bool) -> str: + """Same as usual converter, but removes data URIs""" + + alt = el.attrs.get("alt", None) or "" + src = el.attrs.get("src", None) or "" + title = el.attrs.get("title", None) or "" + title_part = ' "%s"' % title.replace('"', r"\"") if title else "" + if convert_as_inline and el.parent.name not in self.options["keep_inline_images_in"]: + return alt + + # Remove dataURIs + if src.startswith("data:"): + src = src.split(",")[0] + "..." + + return "![%s](%s%s)" % (alt, src, title_part) + + def convert_soup(self, soup: Any) -> str: + return super().convert_soup(soup) # type: ignore + + +class DocumentConverterResult: + """The result of converting a document to text.""" + + def __init__(self, title: Union[str, None] = None, text_content: str = ""): + self.title: Union[str, None] = title + self.text_content: str = text_content + + +class DocumentConverter: + """Abstract superclass of all DocumentConverters.""" + + def convert(self, local_path: str, **kwargs: Any) -> Union[None, DocumentConverterResult]: + raise NotImplementedError() + + +class PlainTextConverter(DocumentConverter): + """Anything with content type text/plain""" + + def convert(self, local_path: str, **kwargs: Any) -> Union[None, DocumentConverterResult]: + # Guess the content type from any file extension that might be around + content_type, _ = mimetypes.guess_type("__placeholder" + kwargs.get("file_extension", "")) + + # Only accept text files + if content_type is None: + return None + elif "text/" not in content_type.lower(): + return None + + text_content = "" + with open(local_path, "rt", encoding="utf-8") as fh: + text_content = fh.read() + return DocumentConverterResult( + title=None, + text_content=text_content, + ) + + +class HtmlConverter(DocumentConverter): + """Anything with content type text/html""" + + def convert(self, local_path: str, **kwargs: Any) -> Union[None, DocumentConverterResult]: + # Bail if not html + extension = kwargs.get("file_extension", "") + if extension.lower() not in [".html", ".htm"]: + return None + + result = None + with open(local_path, "rt", encoding="utf-8") as fh: + result = self._convert(fh.read()) + + return result + + def _convert(self, html_content: str) -> Union[None, DocumentConverterResult]: + """Helper function that converts and HTML string.""" + + # Parse the string + soup = BeautifulSoup(html_content, "html.parser") + + # Remove javascript and style blocks + for script in soup(["script", "style"]): + script.extract() + + # Print only the main content + body_elm = soup.find("body") + webpage_text = "" + if body_elm: + webpage_text = _CustomMarkdownify().convert_soup(body_elm) + else: + webpage_text = _CustomMarkdownify().convert_soup(soup) + + assert isinstance(webpage_text, str) + + return DocumentConverterResult( + title=None if soup.title is None else soup.title.string, text_content=webpage_text + ) + + +class WikipediaConverter(DocumentConverter): + """Handle Wikipedia pages separately, focusing only on the main document content.""" + + def convert(self, local_path: str, **kwargs: Any) -> Union[None, DocumentConverterResult]: + # Bail if not Wikipedia + extension = kwargs.get("file_extension", "") + if extension.lower() not in [".html", ".htm"]: + return None + url = kwargs.get("url", "") + if not re.search(r"^https?:\/\/[a-zA-Z]{2,3}\.wikipedia.org\/", url): + return None + + # Parse the file + soup = None + with open(local_path, "rt", encoding="utf-8") as fh: + soup = BeautifulSoup(fh.read(), "html.parser") + + # Remove javascript and style blocks + for script in soup(["script", "style"]): + script.extract() + + # Print only the main content + body_elm = soup.find("div", {"id": "mw-content-text"}) + title_elm = soup.find("span", {"class": "mw-page-title-main"}) + + webpage_text = "" + main_title = None if soup.title is None else soup.title.string + + if body_elm: + # What's the title + if title_elm and len(title_elm) > 0: + main_title = title_elm.string # type: ignore + assert isinstance(main_title, str) + + # Convert the page + webpage_text = f"# {main_title}\n\n" + _CustomMarkdownify().convert_soup(body_elm) + else: + webpage_text = _CustomMarkdownify().convert_soup(soup) + + return DocumentConverterResult( + title=main_title, + text_content=webpage_text, + ) + + +class YouTubeConverter(DocumentConverter): + """Handle YouTube specially, focusing on the video title, description, and transcript.""" + + def convert(self, local_path: str, **kwargs: Any) -> Union[None, DocumentConverterResult]: + # Bail if not YouTube + extension = kwargs.get("file_extension", "") + if extension.lower() not in [".html", ".htm"]: + return None + url = kwargs.get("url", "") + if not url.startswith("https://www.youtube.com/watch?"): + return None + + # Parse the file + soup = None + with open(local_path, "rt", encoding="utf-8") as fh: + soup = BeautifulSoup(fh.read(), "html.parser") + + # Read the meta tags + assert soup.title is not None and soup.title.string is not None + metadata: Dict[str, str] = {"title": soup.title.string} + for meta in soup(["meta"]): + for a in meta.attrs: + if a in ["itemprop", "property", "name"]: + metadata[meta[a]] = meta.get("content", "") + break + + # We can also try to read the full description. This is more prone to breaking, since it reaches into the page implementation + try: + for script in soup(["script"]): + content = script.text + if "ytInitialData" in content: + lines = re.split(r"\r?\n", content) + obj_start = lines[0].find("{") + obj_end = lines[0].rfind("}") + if obj_start >= 0 and obj_end >= 0: + data = json.loads(lines[0][obj_start : obj_end + 1]) + attrdesc = self._findKey(data, "attributedDescriptionBodyText") # type: ignore + if attrdesc: + metadata["description"] = str(attrdesc["content"]) + break + except Exception: + pass + + # Start preparing the page + webpage_text = "# YouTube\n" + + title = self._get(metadata, ["title", "og:title", "name"]) # type: ignore + assert isinstance(title, str) + + if title: + webpage_text += f"\n## {title}\n" + + stats = "" + views = self._get(metadata, ["interactionCount"]) # type: ignore + if views: + stats += f"- **Views:** {views}\n" + + keywords = self._get(metadata, ["keywords"]) # type: ignore + if keywords: + stats += f"- **Keywords:** {keywords}\n" + + runtime = self._get(metadata, ["duration"]) # type: ignore + if runtime: + stats += f"- **Runtime:** {runtime}\n" + + if len(stats) > 0: + webpage_text += f"\n### Video Metadata\n{stats}\n" + + description = self._get(metadata, ["description", "og:description"]) # type: ignore + if description: + webpage_text += f"\n### Description\n{description}\n" + + if IS_YOUTUBE_TRANSCRIPT_CAPABLE: + transcript_text = "" + parsed_url = urlparse(url) # type: ignore + params = parse_qs(parsed_url.query) # type: ignore + if "v" in params: + assert isinstance(params["v"][0], str) + video_id = str(params["v"][0]) + try: + # Must be a single transcript. + transcript = YouTubeTranscriptApi.get_transcript(video_id) # type: ignore + transcript_text = " ".join([part["text"] for part in transcript]) # type: ignore + # Alternative formatting: + # formatter = TextFormatter() + # formatter.format_transcript(transcript) + except Exception: + pass + if transcript_text: + webpage_text += f"\n### Transcript\n{transcript_text}\n" + + title = title if title else soup.title.string + assert isinstance(title, str) + + return DocumentConverterResult( + title=title, + text_content=webpage_text, + ) + + def _get(self, metadata: Dict[str, str], keys: List[str], default: Union[str, None] = None) -> Union[str, None]: + for k in keys: + if k in metadata: + return metadata[k] + return default + + def _findKey(self, json: Any, key: str) -> Union[str, None]: # TODO: Fix json type + if isinstance(json, list): + for elm in json: + ret = self._findKey(elm, key) + if ret is not None: + return ret + elif isinstance(json, dict): + for k in json: + if k == key: + return json[k] + else: + ret = self._findKey(json[k], key) + if ret is not None: + return ret + return None + + +class BingSerpConverter(DocumentConverter): + """ + Handle Bing results pages (only the organic search results). + NOTE: It is better to use the Bing API + """ + + def convert(self, local_path, **kwargs) -> Union[None, DocumentConverterResult]: + # Bail if not a Bing SERP + extension = kwargs.get("file_extension", "") + if extension.lower() not in [".html", ".htm"]: + return None + url = kwargs.get("url", "") + if not re.search(r"^https://www\.bing\.com/search\?q=", url): + return None + + # Parse the query parameters + parsed_params = parse_qs(urlparse(url).query) + query = parsed_params.get("q", [""])[0] + + # Parse the file + soup = None + with open(local_path, "rt", encoding="utf-8") as fh: + soup = BeautifulSoup(fh.read(), "html.parser") + + # Clean up some formatting + for tptt in soup.find_all(class_="tptt"): + if hasattr(tptt, "string") and tptt.string: + tptt.string += " " + for slug in soup.find_all(class_="algoSlug_icon"): + slug.extract() + + # Parse the algorithmic results + _markdownify = _CustomMarkdownify() + results = list() + for result in soup.find_all(class_="b_algo"): + # Rewrite redirect urls + for a in result.find_all("a", href=True): + parsed_href = urlparse(a["href"]) + qs = parse_qs(parsed_href.query) + + # The destination is contained in the u parameter, + # but appears to be base64 encoded, with some prefix + if "u" in qs: + u = qs["u"][0][2:].strip() + "==" # Python 3 doesn't care about extra padding + + try: + # RFC 4648 / Base64URL" variant, which uses "-" and "_" + a["href"] = base64.b64decode(u, altchars="-_").decode("utf-8") + except UnicodeDecodeError: + pass + except binascii.Error: + pass + + # Convert to markdown + md_result = _markdownify.convert_soup(result).strip() + lines = [line.strip() for line in re.split(r"\n+", md_result)] + results.append("\n".join([line for line in lines if len(line) > 0])) + + webpage_text = f"## A Bing search for '{query}' found the following results:\n\n" + "\n\n".join(results) + + return DocumentConverterResult( + title=None if soup.title is None else soup.title.string, + text_content=webpage_text, + ) + + +class PdfConverter(DocumentConverter): + """ + Converts PDFs to Markdown. Most style information is ignored, so the results are essentially plain-text. + """ + + def convert(self, local_path, **kwargs) -> Union[None, DocumentConverterResult]: + # Bail if not a PDF + extension = kwargs.get("file_extension", "") + if extension.lower() != ".pdf": + return None + converter = DocLingDocumentConverter() + result = converter.convert(local_path) + text_content = result.document.export_to_markdown() + return DocumentConverterResult( + title=None, + # text_content=pdfminer.high_level.extract_text(local_path), + text_content=text_content + ) + + +class DocxConverter(HtmlConverter): + """ + Converts DOCX files to Markdown. Style information (e.g.m headings) and tables are preserved where possible. + """ + + def convert(self, local_path, **kwargs) -> Union[None, DocumentConverterResult]: + # Bail if not a DOCX + extension = kwargs.get("file_extension", "") + if extension.lower() != ".docx": + return None + + result = None + with open(local_path, "rb") as docx_file: + result = mammoth.convert_to_html(docx_file) + html_content = result.value + result = self._convert(html_content) + + return result + + +class XlsxConverter(HtmlConverter): + """ + Converts XLSX files to Markdown, with each sheet presented as a separate Markdown table. + """ + + def convert(self, local_path, **kwargs) -> Union[None, DocumentConverterResult]: + # Bail if not a XLSX + extension = kwargs.get("file_extension", "") + if extension.lower() != ".xlsx": + return None + + sheets = pd.read_excel(local_path, sheet_name=None) + md_content = "" + for s in sheets: + md_content += f"## {s}\n" + html_content = sheets[s].to_html(index=False) + md_content += self._convert(html_content).text_content.strip() + "\n\n" + + return DocumentConverterResult( + title=None, + text_content=md_content.strip(), + ) + + +class PptxConverter(HtmlConverter): + """ + Converts PPTX files to Markdown. Supports heading, tables and images with alt text. + """ + + def convert(self, local_path, **kwargs) -> Union[None, DocumentConverterResult]: + # Bail if not a PPTX + extension = kwargs.get("file_extension", "") + if extension.lower() != ".pptx": + return None + + md_content = "" + + presentation = pptx.Presentation(local_path) + slide_num = 0 + for slide in presentation.slides: + slide_num += 1 + + md_content += f"\n\n\n" + + title = slide.shapes.title + for shape in slide.shapes: + # Pictures + if self._is_picture(shape): + # https://github.com/scanny/python-pptx/pull/512#issuecomment-1713100069 + alt_text = "" + try: + alt_text = shape._element._nvXxPr.cNvPr.attrib.get("descr", "") + except Exception: + pass + + # A placeholder name + filename = re.sub(r"\W", "", shape.name) + ".jpg" + md_content += "\n![" + (alt_text if alt_text else shape.name) + "](" + filename + ")\n" + + # Tables + if self._is_table(shape): + html_table = "" + first_row = True + for row in shape.table.rows: + html_table += "" + for cell in row.cells: + if first_row: + html_table += "" + else: + html_table += "" + html_table += "" + first_row = False + html_table += "
" + html.escape(cell.text) + "" + html.escape(cell.text) + "
" + md_content += "\n" + self._convert(html_table).text_content.strip() + "\n" + + # Text areas + elif shape.has_text_frame: + if shape == title: + md_content += "# " + shape.text.lstrip() + "\n" + else: + md_content += shape.text + "\n" + + md_content = md_content.strip() + + if slide.has_notes_slide: + md_content += "\n\n### Notes:\n" + notes_frame = slide.notes_slide.notes_text_frame + if notes_frame is not None: + md_content += notes_frame.text + md_content = md_content.strip() + + return DocumentConverterResult( + title=None, + text_content=md_content.strip(), + ) + + def _is_picture(self, shape): + if shape.shape_type == pptx.enum.shapes.MSO_SHAPE_TYPE.PICTURE: + return True + if shape.shape_type == pptx.enum.shapes.MSO_SHAPE_TYPE.PLACEHOLDER: + if hasattr(shape, "image"): + return True + return False + + def _is_table(self, shape): + if shape.shape_type == pptx.enum.shapes.MSO_SHAPE_TYPE.TABLE: + return True + return False + + +class MediaConverter(DocumentConverter): + """ + Abstract class for multi-modal media (e.g., images and audio) + """ + + def _get_metadata(self, local_path): + exiftool = shutil.which("exiftool") + if not exiftool: + return None + else: + try: + result = subprocess.run([exiftool, "-json", local_path], capture_output=True, text=True).stdout + return json.loads(result)[0] + except Exception: + return None + + +class WavConverter(MediaConverter): + """ + Converts WAV files to markdown via extraction of metadata (if `exiftool` is installed), and speech transcription (if `speech_recognition` is installed). + """ + + def convert(self, local_path, **kwargs) -> Union[None, DocumentConverterResult]: + # Bail if not a XLSX + extension = kwargs.get("file_extension", "") + if extension.lower() != ".wav": + return None + + md_content = "" + + # Add metadata + metadata = self._get_metadata(local_path) + if metadata: + for f in [ + "Title", + "Artist", + "Author", + "Band", + "Album", + "Genre", + "Track", + "DateTimeOriginal", + "CreateDate", + "Duration", + ]: + if f in metadata: + md_content += f"{f}: {metadata[f]}\n" + + # Transcribe + if IS_AUDIO_TRANSCRIPTION_CAPABLE: + try: + transcript = self._transcribe_audio(local_path) + md_content += "\n\n### Audio Transcript:\n" + ( + "[No speech detected]" if transcript == "" else transcript + ) + except Exception: + md_content += "\n\n### Audio Transcript:\nError. Could not transcribe this audio." + + return DocumentConverterResult( + title=None, + text_content=md_content.strip(), + ) + + def _transcribe_audio(self, local_path) -> str: + recognizer = sr.Recognizer() + with sr.AudioFile(local_path) as source: + audio = recognizer.record(source) + return recognizer.recognize_google(audio).strip() + + +class Mp3Converter(WavConverter): + """ + Converts MP3 files to markdown via extraction of metadata (if `exiftool` is installed), and speech transcription (if `speech_recognition` AND `pydub` are installed). + """ + + def convert(self, local_path, **kwargs) -> Union[None, DocumentConverterResult]: + # Bail if not a MP3 + extension = kwargs.get("file_extension", "") + if extension.lower() != ".mp3": + return None + + md_content = "" + + # Add metadata + metadata = self._get_metadata(local_path) + if metadata: + for f in [ + "Title", + "Artist", + "Author", + "Band", + "Album", + "Genre", + "Track", + "DateTimeOriginal", + "CreateDate", + "Duration", + ]: + if f in metadata: + md_content += f"{f}: {metadata[f]}\n" + + # Transcribe + if IS_AUDIO_TRANSCRIPTION_CAPABLE: + handle, temp_path = tempfile.mkstemp(suffix=".wav") + os.close(handle) + try: + sound = pydub.AudioSegment.from_mp3(local_path) + sound.export(temp_path, format="wav") + + _args = dict() + _args.update(kwargs) + _args["file_extension"] = ".wav" + + try: + transcript = super()._transcribe_audio(temp_path).strip() + md_content += "\n\n### Audio Transcript:\n" + ( + "[No speech detected]" if transcript == "" else transcript + ) + except Exception: + md_content += "\n\n### Audio Transcript:\nError. Could not transcribe this audio." + + finally: + os.unlink(temp_path) + + # Return the result + return DocumentConverterResult( + title=None, + text_content=md_content.strip(), + ) + + +class ImageConverter(MediaConverter): + """ + Converts images to markdown via extraction of metadata (if `exiftool` is installed), OCR (if `easyocr` is installed), and description via a multimodal LLM (if an mlm_client is configured). + """ + + def convert(self, local_path, **kwargs) -> Union[None, DocumentConverterResult]: + # Bail if not a XLSX + extension = kwargs.get("file_extension", "") + if extension.lower() not in [".jpg", ".jpeg", ".png"]: + return None + + md_content = "" + + # Add metadata + metadata = self._get_metadata(local_path) + if metadata: + for f in [ + "ImageSize", + "Title", + "Caption", + "Description", + "Keywords", + "Artist", + "Author", + "DateTimeOriginal", + "CreateDate", + "GPSPosition", + ]: + if f in metadata: + md_content += f"{f}: {metadata[f]}\n" + + # Try describing the image with GPTV + mlm_client = kwargs.get("mlm_client") + mlm_model = kwargs.get("mlm_model") + if mlm_client is not None and mlm_model is not None: + md_content += ( + "\n# Description:\n" + + self._get_mlm_description( + local_path, extension, mlm_client, mlm_model, prompt=kwargs.get("mlm_prompt") + ).strip() + + "\n" + ) + + return DocumentConverterResult( + title=None, + text_content=md_content, + ) + + def _get_mlm_description(self, local_path, extension, client, model, prompt=None): + if prompt is None or prompt.strip() == "": + prompt = "Write a detailed caption for this image." + + sys.stderr.write(f"MLM Prompt:\n{prompt}\n") + + data_uri = "" + with open(local_path, "rb") as image_file: + content_type, encoding = mimetypes.guess_type("_dummy" + extension) + if content_type is None: + content_type = "image/jpeg" + image_base64 = base64.b64encode(image_file.read()).decode("utf-8") + data_uri = f"data:{content_type};base64,{image_base64}" + + messages = [ + { + "role": "user", + "content": [ + {"type": "text", "text": prompt}, + { + "type": "image_url", + "image_url": { + "url": data_uri, + }, + }, + ], + } + ] + + response = client.chat.completions.create(model=model, messages=messages) + return response.choices[0].message.content + + +class FileConversionException(BaseException): + pass + + +class UnsupportedFormatException(BaseException): + pass + + +class MarkdownConverter: + """(In preview) An extremely simple text-based document reader, suitable for LLM use. + This reader will convert common file-types or webpages to Markdown.""" + + def __init__( + self, + requests_session: Optional[requests.Session] = None, + mlm_client: Optional[Any] = None, + mlm_model: Optional[Any] = None, + ): + if requests_session is None: + self._requests_session = requests.Session() + else: + self._requests_session = requests_session + + self._mlm_client = mlm_client + self._mlm_model = mlm_model + + self._page_converters: List[DocumentConverter] = [] + + # Register converters for successful browsing operations + # Later registrations are tried first / take higher priority than earlier registrations + # To this end, the most specific converters should appear below the most generic converters + self.register_page_converter(PlainTextConverter()) + self.register_page_converter(HtmlConverter()) + self.register_page_converter(WikipediaConverter()) + self.register_page_converter(YouTubeConverter()) + self.register_page_converter(BingSerpConverter()) + self.register_page_converter(DocxConverter()) + self.register_page_converter(XlsxConverter()) + self.register_page_converter(PptxConverter()) + self.register_page_converter(WavConverter()) + self.register_page_converter(Mp3Converter()) + self.register_page_converter(ImageConverter()) + self.register_page_converter(PdfConverter()) + + def convert( + self, source: Union[str, requests.Response], **kwargs: Any + ) -> DocumentConverterResult: # TODO: deal with kwargs + """ + Args: + - source: can be a string representing a path or url, or a requests.response object + - extension: specifies the file extension to use when interpreting the file. If None, infer from source (path, uri, content-type, etc.) + """ + + # Local path or url + if isinstance(source, str): + if source.startswith("http://") or source.startswith("https://") or source.startswith("file://"): + return self.convert_url(source, **kwargs) + else: + return self.convert_local(source, **kwargs) + # Request response + elif isinstance(source, requests.Response): + return self.convert_response(source, **kwargs) + + def convert_local(self, path: str, **kwargs: Any) -> DocumentConverterResult: # TODO: deal with kwargs + # Prepare a list of extensions to try (in order of priority) + ext = kwargs.get("file_extension") + extensions = [ext] if ext is not None else [] + + # Get extension alternatives from the path and puremagic + base, ext = os.path.splitext(path) + self._append_ext(extensions, ext) + self._append_ext(extensions, self._guess_ext_magic(path)) + + # Convert + return self._convert(path, extensions, **kwargs) + + # TODO what should stream's type be? + def convert_stream(self, stream: Any, **kwargs: Any) -> DocumentConverterResult: # TODO: deal with kwargs + # Prepare a list of extensions to try (in order of priority) + ext = kwargs.get("file_extension") + extensions = [ext] if ext is not None else [] + + # Save the file locally to a temporary file. It will be deleted before this method exits + handle, temp_path = tempfile.mkstemp() + fh = os.fdopen(handle, "wb") + result = None + try: + # Write to the temporary file + content = stream.read() + if isinstance(content, str): + fh.write(content.encode("utf-8")) + else: + fh.write(content) + fh.close() + + # Use puremagic to check for more extension options + self._append_ext(extensions, self._guess_ext_magic(temp_path)) + + # Convert + result = self._convert(temp_path, extensions, **kwargs) + # Clean up + finally: + try: + fh.close() + except Exception: + pass + os.unlink(temp_path) + + return result + + def convert_url(self, url: str, **kwargs: Any) -> DocumentConverterResult: # TODO: fix kwargs type + # Send a HTTP request to the URL + response = self._requests_session.get(url, stream=True) + response.raise_for_status() + return self.convert_response(response, **kwargs) + + def convert_response( + self, response: requests.Response, **kwargs: Any + ) -> DocumentConverterResult: # TODO fix kwargs type + # Prepare a list of extensions to try (in order of priority) + ext = kwargs.get("file_extension") + extensions = [ext] if ext is not None else [] + + # Guess from the mimetype + content_type = response.headers.get("content-type", "").split(";")[0] + self._append_ext(extensions, mimetypes.guess_extension(content_type)) + + # Read the content disposition if there is one + content_disposition = response.headers.get("content-disposition", "") + m = re.search(r"filename=([^;]+)", content_disposition) + if m: + base, ext = os.path.splitext(m.group(1).strip("\"'")) + self._append_ext(extensions, ext) + + # Read from the extension from the path + base, ext = os.path.splitext(urlparse(response.url).path) + self._append_ext(extensions, ext) + + # Save the file locally to a temporary file. It will be deleted before this method exits + handle, temp_path = tempfile.mkstemp() + fh = os.fdopen(handle, "wb") + result = None + try: + # Download the file + for chunk in response.iter_content(chunk_size=512): + fh.write(chunk) + fh.close() + + # Use puremagic to check for more extension options + self._append_ext(extensions, self._guess_ext_magic(temp_path)) + + # Convert + result = self._convert(temp_path, extensions, url=response.url) + # Clean up + finally: + try: + fh.close() + except Exception: + pass + os.unlink(temp_path) + + return result + + def _convert(self, local_path: str, extensions: List[Union[str, None]], **kwargs) -> DocumentConverterResult: + error_trace = "" + for ext in extensions + [None]: # Try last with no extension + for converter in self._page_converters: + _kwargs = copy.deepcopy(kwargs) + + # Overwrite file_extension appropriately + if ext is None: + if "file_extension" in _kwargs: + del _kwargs["file_extension"] + else: + _kwargs.update({"file_extension": ext}) + + # Copy any additional global options + if "mlm_client" not in _kwargs and self._mlm_client is not None: + _kwargs["mlm_client"] = self._mlm_client + + if "mlm_model" not in _kwargs and self._mlm_model is not None: + _kwargs["mlm_model"] = self._mlm_model + + # If we hit an error log it and keep trying + # try: + res = converter.convert(local_path, **_kwargs) + # except Exception: + # error_trace = ("\n\n" + traceback.format_exc()).strip() + + if res is not None: + # Normalize the content + res.text_content = "\n".join([line.rstrip() for line in re.split(r"\r?\n", res.text_content)]) + res.text_content = re.sub(r"\n{3,}", "\n\n", res.text_content) + + # Todo + return res + + # If we got this far without success, report any exceptions + if len(error_trace) > 0: + raise FileConversionException( + f"Could not convert '{local_path}' to Markdown. File type was recognized as {extensions}. While converting the file, the following error was encountered:\n\n{error_trace}" + ) + + # Nothing can handle it! + raise UnsupportedFormatException( + f"Could not convert '{local_path}' to Markdown. The formats {extensions} are not supported." + ) + + def _append_ext(self, extensions, ext): + """Append a unique non-None, non-empty extension to a list of extensions.""" + if ext is None: + return + ext = ext.strip() + if ext == "": + return + # if ext not in extensions: + if True: + extensions.append(ext) + + def _guess_ext_magic(self, path): + """Use puremagic (a Python implementation of libmagic) to guess a file's extension based on the first few bytes.""" + # Use puremagic to guess + try: + guesses = puremagic.magic_file(path) + if len(guesses) > 0: + ext = guesses[0].extension.strip() + if len(ext) > 0: + return ext + except FileNotFoundError: + pass + except IsADirectoryError: + pass + except PermissionError: + pass + return None + + def register_page_converter(self, converter: DocumentConverter) -> None: + """Register a page text converter.""" + self._page_converters.insert(0, converter) diff --git a/autoagent/environment/markdown_browser/requests_markdown_browser.py b/autoagent/environment/markdown_browser/requests_markdown_browser.py new file mode 100644 index 0000000..b8b17c6 --- /dev/null +++ b/autoagent/environment/markdown_browser/requests_markdown_browser.py @@ -0,0 +1,454 @@ +# ruff: noqa: E722 +import datetime +import html +import io +import mimetypes +import os +import pathlib +import re +import time +import traceback +import uuid +from typing import Any, Dict, List, Optional, Tuple, Union +from urllib.parse import unquote, urljoin, urlparse + +import pathvalidate +import requests + +from .abstract_markdown_browser import AbstractMarkdownBrowser +from .markdown_search import AbstractMarkdownSearch, BingMarkdownSearch + +# TODO: Fix unfollowed import +from .mdconvert import FileConversionException, MarkdownConverter, UnsupportedFormatException # type: ignore + + +class RequestsMarkdownBrowser(AbstractMarkdownBrowser): + """ + (In preview) An extremely simple Python requests-powered Markdown web browser. + This browser cannot run JavaScript, compute CSS, etc. It simply fetches the HTML document, and converts it to Markdown. + See AbstractMarkdownBrowser for more details. + """ + + # TODO: Fix unfollowed import + def __init__( # type: ignore + self, + local_root: str, + workplace_name: str, + start_page: Union[str, None] = None, + viewport_size: Union[int, None] = 1024 * 8, + downloads_folder: Union[str, None] = None, + search_engine: Union[AbstractMarkdownSearch, None] = None, + markdown_converter: Union[MarkdownConverter, None] = None, + requests_session: Union[requests.Session, None] = None, + requests_get_kwargs: Union[Dict[str, Any], None] = None, + ): + """ + Instantiate a new RequestsMarkdownBrowser. + + Arguments: + start_page: The page on which the browser starts (default: "about:blank") + viewport_size: Approximately how many *characters* fit in the viewport. Viewport dimensions are adjusted dynamically to avoid cutting off words (default: 8192). + downloads_folder: Path to where downloads are saved. If None, downloads are disabled. (default: None) + search_engine: An instance of MarkdownSearch, which handles web searches performed by this browser (default: a new `BingMarkdownSearch()` with default parameters) + markdown_converted: An instance of a MarkdownConverter used to convert HTML pages and downloads to Markdown (default: a new `MarkdownConerter()` with default parameters) + request_session: The session from which to issue requests (default: a new `requests.Session()` instance with default parameters) + request_get_kwargs: Extra parameters passed to evert `.get()` call made to requests. + """ + self.local_workplace = os.path.join(local_root, workplace_name) + self.docker_workplace = f"/{workplace_name}" + self.start_page: str = start_page if start_page else "about:blank" + self.viewport_size = viewport_size # Applies only to the standard uri types + self.downloads_folder = downloads_folder + self.history: List[Tuple[str, float]] = list() + self.page_title: Optional[str] = None + self.viewport_current_page = 0 + self.viewport_pages: List[Tuple[int, int]] = list() + self.set_address(self.start_page) + self._page_content: str = "" + + if search_engine is None: + self._search_engine: AbstractMarkdownSearch = BingMarkdownSearch() + else: + self._search_engine = search_engine + + if markdown_converter is None: + self._markdown_converter = MarkdownConverter() + else: + self._markdown_converter = markdown_converter + + if requests_session is None: + self._requests_session = requests.Session() + else: + self._requests_session = requests_session + + if requests_get_kwargs is None: + self._requests_get_kwargs = {} + else: + self._requests_get_kwargs = requests_get_kwargs + + self._find_on_page_query: Union[str, None] = None + self._find_on_page_last_result: Union[int, None] = None # Location of the last result + + @property + def address(self) -> str: + """Return the address of the current page.""" + return self.history[-1][0] + def _convert_docker_to_local(self, path: str) -> str: + assert self.docker_workplace in path, f"The path must be a absolute path from `{self.docker_workplace}/` directory" + local_path = path.replace(self.docker_workplace, self.local_workplace) + return local_path + def _convert_local_to_docker(self, path: str) -> str: + assert self.local_workplace in path, f"The path must be a absolute path from `{self.local_workplace}/` directory" + docker_path = path.replace(self.local_workplace, self.docker_workplace) + return docker_path + + def set_address(self, uri_or_path: str) -> None: + """Sets the address of the current page. + This will result in the page being fetched via the underlying requests session. + + Arguments: + uri_or_path: The fully-qualified URI to fetch, or the path to fetch from the current location. If the URI protocol is `search:`, the remainder of the URI is interpreted as a search query, and a web search is performed. If the URI protocol is `file://`, the remainder of the URI is interpreted as a local absolute file path. + """ + # TODO: Handle anchors + self.history.append((uri_or_path, time.time())) + + # Handle special URIs + if uri_or_path == "about:blank": + self._set_page_content("") + elif uri_or_path.startswith("search:"): + query = uri_or_path[len("search:") :].strip() + results = self._search_engine.search(query) + self.page_title = f"{query} - Search" + self._set_page_content(results, split_pages=False) + else: + if ( + not uri_or_path.startswith("http:") + and not uri_or_path.startswith("https:") + and not uri_or_path.startswith("file:") + ): + if len(self.history) > 1: + prior_address = self.history[-2][0] + uri_or_path = urljoin(prior_address, uri_or_path) + # Update the address with the fully-qualified path + self.history[-1] = (uri_or_path, self.history[-1][1]) + self._fetch_page(uri_or_path) + + self.viewport_current_page = 0 + self.find_on_page_query = None + self.find_on_page_viewport = None + + @property + def viewport(self) -> str: + """Return the content of the current viewport.""" + bounds = self.viewport_pages[self.viewport_current_page] + return self.page_content[bounds[0] : bounds[1]] + + @property + def page_content(self) -> str: + """Return the full contents of the current page.""" + return self._page_content + + def _set_page_content(self, content: str, split_pages: bool = True) -> None: + """Sets the text content of the current page.""" + self._page_content = content + + if split_pages: + self._split_pages() + else: + self.viewport_pages = [(0, len(self._page_content))] + + if self.viewport_current_page >= len(self.viewport_pages): + self.viewport_current_page = len(self.viewport_pages) - 1 + + def page_down(self) -> None: + """Move the viewport down one page, if possible.""" + self.viewport_current_page = min(self.viewport_current_page + 1, len(self.viewport_pages) - 1) + + def page_up(self) -> None: + """Move the viewport up one page, if possible.""" + self.viewport_current_page = max(self.viewport_current_page - 1, 0) + + def page_to(self, page_idx: int) -> None: + """Move the viewport to the specified page index.""" + self.viewport_current_page = min(max(page_idx, 0), len(self.viewport_pages) - 1) + + def find_on_page(self, query: str) -> Union[str, None]: + """Searches for the query from the current viewport forward, looping back to the start if necessary.""" + + # Did we get here via a previous find_on_page search with the same query? + # If so, map to find_next + if query == self._find_on_page_query and self.viewport_current_page == self._find_on_page_last_result: + return self.find_next() + + # Ok it's a new search start from the current viewport + self._find_on_page_query = query + viewport_match = self._find_next_viewport(query, self.viewport_current_page) + if viewport_match is None: + self._find_on_page_last_result = None + return None + else: + self.viewport_current_page = viewport_match + self._find_on_page_last_result = viewport_match + return self.viewport + + def find_next(self) -> Union[str, None]: + """Scroll to the next viewport that matches the query""" + + if self._find_on_page_query is None: + return None + + starting_viewport = self._find_on_page_last_result + if starting_viewport is None: + starting_viewport = 0 + else: + starting_viewport += 1 + if starting_viewport >= len(self.viewport_pages): + starting_viewport = 0 + + viewport_match = self._find_next_viewport(self._find_on_page_query, starting_viewport) + if viewport_match is None: + self._find_on_page_last_result = None + return None + else: + self.viewport_current_page = viewport_match + self._find_on_page_last_result = viewport_match + return self.viewport + + def _find_next_viewport(self, query: Optional[str], starting_viewport: int) -> Union[int, None]: + """Search for matches between the starting viewport looping when reaching the end.""" + + if query is None: + return None + + # Normalize the query, and convert to a regular expression + nquery = re.sub(r"\*", "__STAR__", query) + nquery = " " + (" ".join(re.split(r"\W+", nquery))).strip() + " " + nquery = nquery.replace(" __STAR__ ", "__STAR__ ") # Merge isolated stars with prior word + nquery = nquery.replace("__STAR__", ".*").lower() + + if nquery.strip() == "": + return None + + idxs: List[int] = list() + idxs.extend(range(starting_viewport, len(self.viewport_pages))) + idxs.extend(range(0, starting_viewport)) + + for i in idxs: + bounds = self.viewport_pages[i] + content = self.page_content[bounds[0] : bounds[1]] + + # TODO: Remove markdown links and images + ncontent = " " + (" ".join(re.split(r"\W+", content))).strip().lower() + " " + if re.search(nquery, ncontent): + return i + + return None + + def visit_page(self, path_or_uri: str) -> str: + """Update the address, visit the page, and return the content of the viewport.""" + self.set_address(path_or_uri) + return self.viewport + + def open_local_file(self, local_path: str) -> str: + """Convert a local file path to a file:/// URI, update the address, visit the page, and return the contents of the viewport.""" + full_path = os.path.abspath(os.path.expanduser(local_path)) + self.set_address(pathlib.Path(full_path).as_uri()) + return self.viewport + + def _split_pages(self) -> None: + """Split the page contents into pages that are approximately the viewport size. Small deviations are permitted to ensure words are not broken.""" + # Handle empty pages + if len(self._page_content) == 0: + self.viewport_pages = [(0, 0)] + return + + # Break the viewport into pages + self.viewport_pages = [] + start_idx = 0 + while start_idx < len(self._page_content): + end_idx = min(start_idx + self.viewport_size, len(self._page_content)) # type: ignore[operator] + # Adjust to end on a space + while end_idx < len(self._page_content) and self._page_content[end_idx - 1] not in [" ", "\t", "\r", "\n"]: + end_idx += 1 + self.viewport_pages.append((start_idx, end_idx)) + start_idx = end_idx + + def _fetch_page( + self, + url: str, + session: Optional[requests.Session] = None, + requests_get_kwargs: Union[Dict[str, Any], None] = None, + ) -> None: + """Fetch a page using the requests library. Then convert it to Markdown, and set `page_content` (which splits the content into pages as necessary. + + Arguments: + url: The fully-qualified URL to fetch. + session: Used to override the session used for this request. If None, use `self._requests_session` as usual. + requests_get_kwargs: Extra arguments passes to `requests.Session.get`. + """ + download_path: str = "" + response: Union[requests.Response, None] = None + # print(url) + try: + if url.startswith("file://"): + download_path = os.path.normcase(os.path.normpath(unquote(url[7:]))) + if os.path.isdir(download_path): # TODO: Fix markdown_converter types + res = self._markdown_converter.convert_stream( # type: ignore + io.StringIO(self._fetch_local_dir(download_path)), file_extension=".html" + ) + self.page_title = res.title + self._set_page_content( + res.text_content, split_pages=False + ) # Like search results, don't split directory listings + else: + res = self._markdown_converter.convert_local(download_path) + self.page_title = res.title + self._set_page_content(res.text_content) + else: + # Send a HTTP request to the URL + if session is None: + session = self._requests_session + + _get_kwargs: Dict[str, Any] = {} # TODO: Deal with kwargs + _get_kwargs.update(self._requests_get_kwargs) + if requests_get_kwargs is not None: + _get_kwargs.update(requests_get_kwargs) + _get_kwargs["stream"] = True + + response = session.get(url, **_get_kwargs) + response.raise_for_status() + + # If the HTTP request was successful + content_type = response.headers.get("content-type", "") + + # Text or HTML + if "text/" in content_type.lower(): + res = self._markdown_converter.convert_response(response) + self.page_title = res.title + self._set_page_content(res.text_content) + # A download + else: + # Was a downloads folder configured? + if self.downloads_folder is None: + self.page_title = "Error 400" + self._set_page_content("## Error 400\n\nClient does not support downloads") + return + + assert self.downloads_folder is not None + + # Try producing a safe filename + fname: str = "" + try: + fname = pathvalidate.sanitize_filename(os.path.basename(urlparse(url).path)).strip() + download_path = os.path.abspath(os.path.join(self.downloads_folder, fname)) + + suffix = 0 + while os.path.exists(download_path) and suffix < 1000: + suffix += 1 + base, ext = os.path.splitext(fname) + new_fname = f"{base}__{suffix}{ext}" + download_path = os.path.abspath(os.path.join(self.downloads_folder, new_fname)) + + except NameError: + pass + + # No suitable name, so make one + if fname == "": + extension = mimetypes.guess_extension(content_type) + if extension is None: + extension = ".download" + fname = str(uuid.uuid4()) + extension + download_path = os.path.abspath(os.path.join(self.downloads_folder, fname)) + + # Open a file for writing + with open(download_path, "wb") as fh: + for chunk in response.iter_content(chunk_size=512): + fh.write(chunk) + + # Render it + local_uri = pathlib.Path(download_path).as_uri() + self.set_address(local_uri) + + except UnsupportedFormatException: + self.page_title = "Download complete." + self._set_page_content(f"# Download complete\n\nSaved file to '{download_path}'") + except FileConversionException: + self.page_title = "Download complete." + self._set_page_content(f"# Download complete\n\nSaved file to '{download_path}'") + except FileNotFoundError: + self.page_title = "Error 404" + self._set_page_content(f"## Error 404\n\nFile not found: {download_path}") + except requests.exceptions.RequestException: + if response is None: + self.page_title = "Request Exception" + self._set_page_content("## Unhandled Request Exception:\n\n" + traceback.format_exc()) + else: + self.page_title = f"Error {response.status_code}" + + # If the error was rendered in HTML we might as well render it + content_type = response.headers.get("content-type", "") + if "text/html" in content_type.lower(): + res = self._markdown_converter.convert(response) + self.page_title = f"Error {response.status_code}" + self._set_page_content(f"## Error {response.status_code}\n\n{res.text_content}") + else: + text = "" + for chunk in response.iter_content(chunk_size=512, decode_unicode=True): + text += chunk + self.page_title = f"Error {response.status_code}" + self._set_page_content(f"## Error {response.status_code}\n\n{text}") + + def _fetch_local_dir(self, local_path: str) -> str: + """Render a local directory listing in HTML to assist with local file browsing via the "file://" protocol. + Through rendered in HTML, later parts of the pipeline will convert the listing to Markdown. + + Arguments: + local_path: A path to the local directory whose contents are to be listed. + + Returns: + A directory listing, rendered in HTML. + """ + pardir = os.path.normpath(os.path.join(local_path, os.pardir)) + pardir_uri = pathlib.Path(pardir).as_uri() + listing = f""" + + + + Index of {html.escape(local_path)} + + +

Index of {html.escape(local_path)}

+ + .. (parent directory) + + + + + +""" + + for entry in os.listdir(local_path): + full_path = os.path.normpath(os.path.join(local_path, entry)) + full_path_uri = pathlib.Path(full_path).as_uri() + size = "" + mtime = datetime.datetime.fromtimestamp(os.path.getmtime(full_path)).strftime("%Y-%m-%d %H:%M") + + if os.path.isdir(full_path): + entry = entry + os.path.sep + else: + size = str(os.path.getsize(full_path)) + + listing += ( + "\n" + + f'' + + f"" + + f"" + + "" + ) + + listing += """ +
NameSizeDate modified
{html.escape(entry)}{html.escape(size)}{html.escape(mtime)}
+ + +""" + return listing diff --git a/autoagent/environment/mdconvert.py b/autoagent/environment/mdconvert.py new file mode 100644 index 0000000..56c14e8 --- /dev/null +++ b/autoagent/environment/mdconvert.py @@ -0,0 +1,1235 @@ + +from browsergym.core.action.functions import goto, page +from autoagent.environment.markdown_browser import MarkdownConverter +def _get_page_markdown(): + """ + Get the markdown content of the current page + + Examples: + _get_page_markdown() + """ + # # type: ignore + import io + import base64 + import binascii + import copy + import html + import json + import mimetypes + import os + import re + import shutil + import subprocess + import sys + import tempfile + from typing import Any, Dict, List, Optional, Union + from urllib.parse import parse_qs, quote, unquote, urlparse, urlunparse + + import mammoth + import markdownify + import pandas as pd + import pdfminer + import pdfminer.high_level + import pptx + + # File-format detection + import puremagic + import requests + from bs4 import BeautifulSoup + + # Optional Transcription support + try: + import pydub + import speech_recognition as sr + + IS_AUDIO_TRANSCRIPTION_CAPABLE = True + except ModuleNotFoundError: + pass + + # Optional YouTube transcription support + try: + from youtube_transcript_api import YouTubeTranscriptApi + + IS_YOUTUBE_TRANSCRIPT_CAPABLE = True + except ModuleNotFoundError: + pass + + + class _CustomMarkdownify(markdownify.MarkdownConverter): + """ + A custom version of markdownify's MarkdownConverter. Changes include: + + - Altering the default heading style to use '#', '##', etc. + - Removing javascript hyperlinks. + - Truncating images with large data:uri sources. + - Ensuring URIs are properly escaped, and do not conflict with Markdown syntax + """ + + def __init__(self, **options: Any): + options["heading_style"] = options.get("heading_style", markdownify.ATX) + # Explicitly cast options to the expected type if necessary + super().__init__(**options) + + def convert_hn(self, n: int, el: Any, text: str, convert_as_inline: bool) -> str: + """Same as usual, but be sure to start with a new line""" + if not convert_as_inline: + if not re.search(r"^\n", text): + return "\n" + super().convert_hn(n, el, text, convert_as_inline) # type: ignore + + return super().convert_hn(n, el, text, convert_as_inline) # type: ignore + + def convert_a(self, el: Any, text: str, convert_as_inline: bool): + """Same as usual converter, but removes Javascript links and escapes URIs.""" + prefix, suffix, text = markdownify.chomp(text) # type: ignore + if not text: + return "" + href = el.get("href") + title = el.get("title") + + # Escape URIs and skip non-http or file schemes + if href: + try: + parsed_url = urlparse(href) # type: ignore + if parsed_url.scheme and parsed_url.scheme.lower() not in ["http", "https", "file"]: # type: ignore + return "%s%s%s" % (prefix, text, suffix) + href = urlunparse(parsed_url._replace(path=quote(unquote(parsed_url.path)))) # type: ignore + except ValueError: # It's not clear if this ever gets thrown + return "%s%s%s" % (prefix, text, suffix) + + # For the replacement see #29: text nodes underscores are escaped + if ( + self.options["autolinks"] + and text.replace(r"\_", "_") == href + and not title + and not self.options["default_title"] + ): + # Shortcut syntax + return "<%s>" % href + if self.options["default_title"] and not title: + title = href + title_part = ' "%s"' % title.replace('"', r"\"") if title else "" + return "%s[%s](%s%s)%s" % (prefix, text, href, title_part, suffix) if href else text + + def convert_img(self, el: Any, text: str, convert_as_inline: bool) -> str: + """Same as usual converter, but removes data URIs""" + + alt = el.attrs.get("alt", None) or "" + src = el.attrs.get("src", None) or "" + title = el.attrs.get("title", None) or "" + title_part = ' "%s"' % title.replace('"', r"\"") if title else "" + if convert_as_inline and el.parent.name not in self.options["keep_inline_images_in"]: + return alt + + # Remove dataURIs + if src.startswith("data:"): + src = src.split(",")[0] + "..." + + return "![%s](%s%s)" % (alt, src, title_part) + + def convert_soup(self, soup: Any) -> str: + return super().convert_soup(soup) # type: ignore + + + class DocumentConverterResult: + """The result of converting a document to text.""" + + def __init__(self, title: Union[str, None] = None, text_content: str = ""): + self.title: Union[str, None] = title + self.text_content: str = text_content + + + class DocumentConverter: + """Abstract superclass of all DocumentConverters.""" + + def convert(self, local_path: str, **kwargs: Any) -> Union[None, DocumentConverterResult]: + raise NotImplementedError() + + + class PlainTextConverter(DocumentConverter): + """Anything with content type text/plain""" + + def convert(self, local_path: str, **kwargs: Any) -> Union[None, DocumentConverterResult]: + # Guess the content type from any file extension that might be around + content_type, _ = mimetypes.guess_type("__placeholder" + kwargs.get("file_extension", "")) + + # Only accept text files + if content_type is None: + return None + elif "text/" not in content_type.lower(): + return None + + text_content = "" + with open(local_path, "rt", encoding="utf-8") as fh: + text_content = fh.read() + return DocumentConverterResult( + title=None, + text_content=text_content, + ) + + + class HtmlConverter(DocumentConverter): + """Anything with content type text/html""" + + def convert(self, local_path: str, **kwargs: Any) -> Union[None, DocumentConverterResult]: + # Bail if not html + extension = kwargs.get("file_extension", "") + if extension.lower() not in [".html", ".htm"]: + return None + + result = None + with open(local_path, "rt", encoding="utf-8") as fh: + result = self._convert(fh.read()) + + return result + + def _convert(self, html_content: str) -> Union[None, DocumentConverterResult]: + """Helper function that converts and HTML string.""" + + # Parse the string + soup = BeautifulSoup(html_content, "html.parser") + + # Remove javascript and style blocks + for script in soup(["script", "style"]): + script.extract() + + # Print only the main content + body_elm = soup.find("body") + webpage_text = "" + if body_elm: + webpage_text = _CustomMarkdownify().convert_soup(body_elm) + else: + webpage_text = _CustomMarkdownify().convert_soup(soup) + + assert isinstance(webpage_text, str) + + return DocumentConverterResult( + title=None if soup.title is None else soup.title.string, text_content=webpage_text + ) + + + class WikipediaConverter(DocumentConverter): + """Handle Wikipedia pages separately, focusing only on the main document content.""" + + def convert(self, local_path: str, **kwargs: Any) -> Union[None, DocumentConverterResult]: + # Bail if not Wikipedia + extension = kwargs.get("file_extension", "") + if extension.lower() not in [".html", ".htm"]: + return None + url = kwargs.get("url", "") + if not re.search(r"^https?:\/\/[a-zA-Z]{2,3}\.wikipedia.org\/", url): + return None + + # Parse the file + soup = None + with open(local_path, "rt", encoding="utf-8") as fh: + soup = BeautifulSoup(fh.read(), "html.parser") + + # Remove javascript and style blocks + for script in soup(["script", "style"]): + script.extract() + + # Print only the main content + body_elm = soup.find("div", {"id": "mw-content-text"}) + title_elm = soup.find("span", {"class": "mw-page-title-main"}) + + webpage_text = "" + main_title = None if soup.title is None else soup.title.string + + if body_elm: + # What's the title + if title_elm and len(title_elm) > 0: + main_title = title_elm.string # type: ignore + assert isinstance(main_title, str) + + # Convert the page + webpage_text = f"# {main_title}\n\n" + _CustomMarkdownify().convert_soup(body_elm) + else: + webpage_text = _CustomMarkdownify().convert_soup(soup) + + return DocumentConverterResult( + title=main_title, + text_content=webpage_text, + ) + + + class YouTubeConverter(DocumentConverter): + """Handle YouTube specially, focusing on the video title, description, and transcript.""" + + def convert(self, local_path: str, **kwargs: Any) -> Union[None, DocumentConverterResult]: + # Bail if not YouTube + extension = kwargs.get("file_extension", "") + if extension.lower() not in [".html", ".htm"]: + return None + url = kwargs.get("url", "") + if not url.startswith("https://www.youtube.com/watch?"): + return None + + # Parse the file + soup = None + with open(local_path, "rt", encoding="utf-8") as fh: + soup = BeautifulSoup(fh.read(), "html.parser") + + # Read the meta tags + assert soup.title is not None and soup.title.string is not None + metadata: Dict[str, str] = {"title": soup.title.string} + for meta in soup(["meta"]): + for a in meta.attrs: + if a in ["itemprop", "property", "name"]: + metadata[meta[a]] = meta.get("content", "") + break + + # We can also try to read the full description. This is more prone to breaking, since it reaches into the page implementation + try: + for script in soup(["script"]): + content = script.text + if "ytInitialData" in content: + lines = re.split(r"\r?\n", content) + obj_start = lines[0].find("{") + obj_end = lines[0].rfind("}") + if obj_start >= 0 and obj_end >= 0: + data = json.loads(lines[0][obj_start : obj_end + 1]) + attrdesc = self._findKey(data, "attributedDescriptionBodyText") # type: ignore + if attrdesc: + metadata["description"] = str(attrdesc["content"]) + break + except Exception: + pass + + # Start preparing the page + webpage_text = "# YouTube\n" + + title = self._get(metadata, ["title", "og:title", "name"]) # type: ignore + assert isinstance(title, str) + + if title: + webpage_text += f"\n## {title}\n" + + stats = "" + views = self._get(metadata, ["interactionCount"]) # type: ignore + if views: + stats += f"- **Views:** {views}\n" + + keywords = self._get(metadata, ["keywords"]) # type: ignore + if keywords: + stats += f"- **Keywords:** {keywords}\n" + + runtime = self._get(metadata, ["duration"]) # type: ignore + if runtime: + stats += f"- **Runtime:** {runtime}\n" + + if len(stats) > 0: + webpage_text += f"\n### Video Metadata\n{stats}\n" + + description = self._get(metadata, ["description", "og:description"]) # type: ignore + if description: + webpage_text += f"\n### Description\n{description}\n" + + if IS_YOUTUBE_TRANSCRIPT_CAPABLE: + transcript_text = "" + parsed_url = urlparse(url) # type: ignore + params = parse_qs(parsed_url.query) # type: ignore + if "v" in params: + assert isinstance(params["v"][0], str) + video_id = str(params["v"][0]) + try: + # Must be a single transcript. + transcript = YouTubeTranscriptApi.get_transcript(video_id) # type: ignore + transcript_text = " ".join([part["text"] for part in transcript]) # type: ignore + # Alternative formatting: + # formatter = TextFormatter() + # formatter.format_transcript(transcript) + except Exception: + pass + if transcript_text: + webpage_text += f"\n### Transcript\n{transcript_text}\n" + + title = title if title else soup.title.string + assert isinstance(title, str) + + return DocumentConverterResult( + title=title, + text_content=webpage_text, + ) + + def _get(self, metadata: Dict[str, str], keys: List[str], default: Union[str, None] = None) -> Union[str, None]: + for k in keys: + if k in metadata: + return metadata[k] + return default + + def _findKey(self, json: Any, key: str) -> Union[str, None]: # TODO: Fix json type + if isinstance(json, list): + for elm in json: + ret = self._findKey(elm, key) + if ret is not None: + return ret + elif isinstance(json, dict): + for k in json: + if k == key: + return json[k] + else: + ret = self._findKey(json[k], key) + if ret is not None: + return ret + return None + + + class BingSerpConverter(DocumentConverter): + """ + Handle Bing results pages (only the organic search results). + NOTE: It is better to use the Bing API + """ + + def convert(self, local_path, **kwargs) -> Union[None, DocumentConverterResult]: + # Bail if not a Bing SERP + extension = kwargs.get("file_extension", "") + if extension.lower() not in [".html", ".htm"]: + return None + url = kwargs.get("url", "") + if not re.search(r"^https://www\.bing\.com/search\?q=", url): + return None + + # Parse the query parameters + parsed_params = parse_qs(urlparse(url).query) + query = parsed_params.get("q", [""])[0] + + # Parse the file + soup = None + with open(local_path, "rt", encoding="utf-8") as fh: + soup = BeautifulSoup(fh.read(), "html.parser") + + # Clean up some formatting + for tptt in soup.find_all(class_="tptt"): + if hasattr(tptt, "string") and tptt.string: + tptt.string += " " + for slug in soup.find_all(class_="algoSlug_icon"): + slug.extract() + + # Parse the algorithmic results + _markdownify = _CustomMarkdownify() + results = list() + for result in soup.find_all(class_="b_algo"): + # Rewrite redirect urls + for a in result.find_all("a", href=True): + parsed_href = urlparse(a["href"]) + qs = parse_qs(parsed_href.query) + + # The destination is contained in the u parameter, + # but appears to be base64 encoded, with some prefix + if "u" in qs: + u = qs["u"][0][2:].strip() + "==" # Python 3 doesn't care about extra padding + + try: + # RFC 4648 / Base64URL" variant, which uses "-" and "_" + a["href"] = base64.b64decode(u, altchars="-_").decode("utf-8") + except UnicodeDecodeError: + pass + except binascii.Error: + pass + + # Convert to markdown + md_result = _markdownify.convert_soup(result).strip() + lines = [line.strip() for line in re.split(r"\n+", md_result)] + results.append("\n".join([line for line in lines if len(line) > 0])) + + webpage_text = f"## A Bing search for '{query}' found the following results:\n\n" + "\n\n".join(results) + + return DocumentConverterResult( + title=None if soup.title is None else soup.title.string, + text_content=webpage_text, + ) + + + class PdfConverter(DocumentConverter): + """ + Converts PDFs to Markdown. Most style information is ignored, so the results are essentially plain-text. + """ + + def convert(self, local_path, **kwargs) -> Union[None, DocumentConverterResult]: + # Bail if not a PDF + extension = kwargs.get("file_extension", "") + if extension.lower() != ".pdf": + return None + + return DocumentConverterResult( + title=None, + text_content=pdfminer.high_level.extract_text(local_path), + ) + + + class DocxConverter(HtmlConverter): + """ + Converts DOCX files to Markdown. Style information (e.g.m headings) and tables are preserved where possible. + """ + + def convert(self, local_path, **kwargs) -> Union[None, DocumentConverterResult]: + # Bail if not a DOCX + extension = kwargs.get("file_extension", "") + if extension.lower() != ".docx": + return None + + result = None + with open(local_path, "rb") as docx_file: + result = mammoth.convert_to_html(docx_file) + html_content = result.value + result = self._convert(html_content) + + return result + + + class XlsxConverter(HtmlConverter): + """ + Converts XLSX files to Markdown, with each sheet presented as a separate Markdown table. + """ + + def convert(self, local_path, **kwargs) -> Union[None, DocumentConverterResult]: + # Bail if not a XLSX + extension = kwargs.get("file_extension", "") + if extension.lower() != ".xlsx": + return None + + sheets = pd.read_excel(local_path, sheet_name=None) + md_content = "" + for s in sheets: + md_content += f"## {s}\n" + html_content = sheets[s].to_html(index=False) + md_content += self._convert(html_content).text_content.strip() + "\n\n" + + return DocumentConverterResult( + title=None, + text_content=md_content.strip(), + ) + + + class PptxConverter(HtmlConverter): + """ + Converts PPTX files to Markdown. Supports heading, tables and images with alt text. + """ + + def convert(self, local_path, **kwargs) -> Union[None, DocumentConverterResult]: + # Bail if not a PPTX + extension = kwargs.get("file_extension", "") + if extension.lower() != ".pptx": + return None + + md_content = "" + + presentation = pptx.Presentation(local_path) + slide_num = 0 + for slide in presentation.slides: + slide_num += 1 + + md_content += f"\n\n\n" + + title = slide.shapes.title + for shape in slide.shapes: + # Pictures + if self._is_picture(shape): + # https://github.com/scanny/python-pptx/pull/512#issuecomment-1713100069 + alt_text = "" + try: + alt_text = shape._element._nvXxPr.cNvPr.attrib.get("descr", "") + except Exception: + pass + + # A placeholder name + filename = re.sub(r"\W", "", shape.name) + ".jpg" + md_content += "\n![" + (alt_text if alt_text else shape.name) + "](" + filename + ")\n" + + # Tables + if self._is_table(shape): + html_table = "" + first_row = True + for row in shape.table.rows: + html_table += "" + for cell in row.cells: + if first_row: + html_table += "" + else: + html_table += "" + html_table += "" + first_row = False + html_table += "
" + html.escape(cell.text) + "" + html.escape(cell.text) + "
" + md_content += "\n" + self._convert(html_table).text_content.strip() + "\n" + + # Text areas + elif shape.has_text_frame: + if shape == title: + md_content += "# " + shape.text.lstrip() + "\n" + else: + md_content += shape.text + "\n" + + md_content = md_content.strip() + + if slide.has_notes_slide: + md_content += "\n\n### Notes:\n" + notes_frame = slide.notes_slide.notes_text_frame + if notes_frame is not None: + md_content += notes_frame.text + md_content = md_content.strip() + + return DocumentConverterResult( + title=None, + text_content=md_content.strip(), + ) + + def _is_picture(self, shape): + if shape.shape_type == pptx.enum.shapes.MSO_SHAPE_TYPE.PICTURE: + return True + if shape.shape_type == pptx.enum.shapes.MSO_SHAPE_TYPE.PLACEHOLDER: + if hasattr(shape, "image"): + return True + return False + + def _is_table(self, shape): + if shape.shape_type == pptx.enum.shapes.MSO_SHAPE_TYPE.TABLE: + return True + return False + + + class MediaConverter(DocumentConverter): + """ + Abstract class for multi-modal media (e.g., images and audio) + """ + + def _get_metadata(self, local_path): + exiftool = shutil.which("exiftool") + if not exiftool: + return None + else: + try: + result = subprocess.run([exiftool, "-json", local_path], capture_output=True, text=True).stdout + return json.loads(result)[0] + except Exception: + return None + + + class WavConverter(MediaConverter): + """ + Converts WAV files to markdown via extraction of metadata (if `exiftool` is installed), and speech transcription (if `speech_recognition` is installed). + """ + + def convert(self, local_path, **kwargs) -> Union[None, DocumentConverterResult]: + # Bail if not a XLSX + extension = kwargs.get("file_extension", "") + if extension.lower() != ".wav": + return None + + md_content = "" + + # Add metadata + metadata = self._get_metadata(local_path) + if metadata: + for f in [ + "Title", + "Artist", + "Author", + "Band", + "Album", + "Genre", + "Track", + "DateTimeOriginal", + "CreateDate", + "Duration", + ]: + if f in metadata: + md_content += f"{f}: {metadata[f]}\n" + + # Transcribe + if IS_AUDIO_TRANSCRIPTION_CAPABLE: + try: + transcript = self._transcribe_audio(local_path) + md_content += "\n\n### Audio Transcript:\n" + ( + "[No speech detected]" if transcript == "" else transcript + ) + except Exception: + md_content += "\n\n### Audio Transcript:\nError. Could not transcribe this audio." + + return DocumentConverterResult( + title=None, + text_content=md_content.strip(), + ) + + def _transcribe_audio(self, local_path) -> str: + recognizer = sr.Recognizer() + with sr.AudioFile(local_path) as source: + audio = recognizer.record(source) + return recognizer.recognize_google(audio).strip() + + + class Mp3Converter(WavConverter): + """ + Converts MP3 files to markdown via extraction of metadata (if `exiftool` is installed), and speech transcription (if `speech_recognition` AND `pydub` are installed). + """ + + def convert(self, local_path, **kwargs) -> Union[None, DocumentConverterResult]: + # Bail if not a MP3 + extension = kwargs.get("file_extension", "") + if extension.lower() != ".mp3": + return None + + md_content = "" + + # Add metadata + metadata = self._get_metadata(local_path) + if metadata: + for f in [ + "Title", + "Artist", + "Author", + "Band", + "Album", + "Genre", + "Track", + "DateTimeOriginal", + "CreateDate", + "Duration", + ]: + if f in metadata: + md_content += f"{f}: {metadata[f]}\n" + + # Transcribe + if IS_AUDIO_TRANSCRIPTION_CAPABLE: + handle, temp_path = tempfile.mkstemp(suffix=".wav") + os.close(handle) + try: + sound = pydub.AudioSegment.from_mp3(local_path) + sound.export(temp_path, format="wav") + + _args = dict() + _args.update(kwargs) + _args["file_extension"] = ".wav" + + try: + transcript = super()._transcribe_audio(temp_path).strip() + md_content += "\n\n### Audio Transcript:\n" + ( + "[No speech detected]" if transcript == "" else transcript + ) + except Exception: + md_content += "\n\n### Audio Transcript:\nError. Could not transcribe this audio." + + finally: + os.unlink(temp_path) + + # Return the result + return DocumentConverterResult( + title=None, + text_content=md_content.strip(), + ) + + + class ImageConverter(MediaConverter): + """ + Converts images to markdown via extraction of metadata (if `exiftool` is installed), OCR (if `easyocr` is installed), and description via a multimodal LLM (if an mlm_client is configured). + """ + + def convert(self, local_path, **kwargs) -> Union[None, DocumentConverterResult]: + # Bail if not a XLSX + extension = kwargs.get("file_extension", "") + if extension.lower() not in [".jpg", ".jpeg", ".png"]: + return None + + md_content = "" + + # Add metadata + metadata = self._get_metadata(local_path) + if metadata: + for f in [ + "ImageSize", + "Title", + "Caption", + "Description", + "Keywords", + "Artist", + "Author", + "DateTimeOriginal", + "CreateDate", + "GPSPosition", + ]: + if f in metadata: + md_content += f"{f}: {metadata[f]}\n" + + # Try describing the image with GPTV + mlm_client = kwargs.get("mlm_client") + mlm_model = kwargs.get("mlm_model") + if mlm_client is not None and mlm_model is not None: + md_content += ( + "\n# Description:\n" + + self._get_mlm_description( + local_path, extension, mlm_client, mlm_model, prompt=kwargs.get("mlm_prompt") + ).strip() + + "\n" + ) + + return DocumentConverterResult( + title=None, + text_content=md_content, + ) + + def _get_mlm_description(self, local_path, extension, client, model, prompt=None): + if prompt is None or prompt.strip() == "": + prompt = "Write a detailed caption for this image." + + sys.stderr.write(f"MLM Prompt:\n{prompt}\n") + + data_uri = "" + with open(local_path, "rb") as image_file: + content_type, encoding = mimetypes.guess_type("_dummy" + extension) + if content_type is None: + content_type = "image/jpeg" + image_base64 = base64.b64encode(image_file.read()).decode("utf-8") + data_uri = f"data:{content_type};base64,{image_base64}" + + messages = [ + { + "role": "user", + "content": [ + {"type": "text", "text": prompt}, + { + "type": "image_url", + "image_url": { + "url": data_uri, + }, + }, + ], + } + ] + + response = client.chat.completions.create(model=model, messages=messages) + return response.choices[0].message.content + + + class FileConversionException(BaseException): + pass + + + class UnsupportedFormatException(BaseException): + pass + + + class MarkdownConverter: + """(In preview) An extremely simple text-based document reader, suitable for LLM use. + This reader will convert common file-types or webpages to Markdown.""" + + def __init__( + self, + requests_session: Optional[requests.Session] = None, + mlm_client: Optional[Any] = None, + mlm_model: Optional[Any] = None, + ): + if requests_session is None: + self._requests_session = requests.Session() + else: + self._requests_session = requests_session + + self._mlm_client = mlm_client + self._mlm_model = mlm_model + + self._page_converters: List[DocumentConverter] = [] + + # Register converters for successful browsing operations + # Later registrations are tried first / take higher priority than earlier registrations + # To this end, the most specific converters should appear below the most generic converters + self.register_page_converter(PlainTextConverter()) + self.register_page_converter(HtmlConverter()) + self.register_page_converter(WikipediaConverter()) + self.register_page_converter(YouTubeConverter()) + # self.register_page_converter(BingSerpConverter()) + self.register_page_converter(DocxConverter()) + self.register_page_converter(XlsxConverter()) + self.register_page_converter(PptxConverter()) + self.register_page_converter(WavConverter()) + self.register_page_converter(Mp3Converter()) + self.register_page_converter(PdfConverter()) + + def convert( + self, source: Union[str, requests.Response], **kwargs: Any + ) -> DocumentConverterResult: # TODO: deal with kwargs + """ + Args: + - source: can be a string representing a path or url, or a requests.response object + - extension: specifies the file extension to use when interpreting the file. If None, infer from source (path, uri, content-type, etc.) + """ + + # Local path or url + if isinstance(source, str): + if source.startswith("http://") or source.startswith("https://") or source.startswith("file://"): + return self.convert_url(source, **kwargs) + else: + return self.convert_local(source, **kwargs) + # Request response + elif isinstance(source, requests.Response): + return self.convert_response(source, **kwargs) + + def convert_local(self, path: str, **kwargs: Any) -> DocumentConverterResult: # TODO: deal with kwargs + # Prepare a list of extensions to try (in order of priority) + ext = kwargs.get("file_extension") + extensions = [ext] if ext is not None else [] + + # Get extension alternatives from the path and puremagic + base, ext = os.path.splitext(path) + self._append_ext(extensions, ext) + self._append_ext(extensions, self._guess_ext_magic(path)) + + # Convert + return self._convert(path, extensions, **kwargs) + + # TODO what should stream's type be? + def convert_stream(self, stream: Any, **kwargs: Any) -> DocumentConverterResult: # TODO: deal with kwargs + # Prepare a list of extensions to try (in order of priority) + ext = kwargs.get("file_extension") + extensions = [ext] if ext is not None else [] + + # Save the file locally to a temporary file. It will be deleted before this method exits + handle, temp_path = tempfile.mkstemp() + fh = os.fdopen(handle, "wb") + result = None + try: + # Write to the temporary file + content = stream.read() + if isinstance(content, str): + fh.write(content.encode("utf-8")) + else: + fh.write(content) + fh.close() + + # Use puremagic to check for more extension options + self._append_ext(extensions, self._guess_ext_magic(temp_path)) + + # Convert + result = self._convert(temp_path, extensions, **kwargs) + # Clean up + finally: + try: + fh.close() + except Exception: + pass + os.unlink(temp_path) + + return result + + def convert_url(self, url: str, **kwargs: Any) -> DocumentConverterResult: # TODO: fix kwargs type + # Send a HTTP request to the URL + response = self._requests_session.get(url, stream=True) + response.raise_for_status() + return self.convert_response(response, **kwargs) + + def convert_response( + self, response: requests.Response, **kwargs: Any + ) -> DocumentConverterResult: # TODO fix kwargs type + # Prepare a list of extensions to try (in order of priority) + ext = kwargs.get("file_extension") + extensions = [ext] if ext is not None else [] + + # Guess from the mimetype + content_type = response.headers.get("content-type", "").split(";")[0] + self._append_ext(extensions, mimetypes.guess_extension(content_type)) + + # Read the content disposition if there is one + content_disposition = response.headers.get("content-disposition", "") + m = re.search(r"filename=([^;]+)", content_disposition) + if m: + base, ext = os.path.splitext(m.group(1).strip("\"'")) + self._append_ext(extensions, ext) + + # Read from the extension from the path + base, ext = os.path.splitext(urlparse(response.url).path) + self._append_ext(extensions, ext) + + # Save the file locally to a temporary file. It will be deleted before this method exits + handle, temp_path = tempfile.mkstemp() + fh = os.fdopen(handle, "wb") + result = None + try: + # Download the file + for chunk in response.iter_content(chunk_size=512): + fh.write(chunk) + fh.close() + + # Use puremagic to check for more extension options + self._append_ext(extensions, self._guess_ext_magic(temp_path)) + + # Convert + result = self._convert(temp_path, extensions, url=response.url) + # Clean up + finally: + try: + fh.close() + except Exception: + pass + os.unlink(temp_path) + + return result + + def _convert(self, local_path: str, extensions: List[Union[str, None]], **kwargs) -> DocumentConverterResult: + error_trace = "" + for ext in extensions + [None]: # Try last with no extension + for converter in self._page_converters: + _kwargs = copy.deepcopy(kwargs) + + # Overwrite file_extension appropriately + if ext is None: + if "file_extension" in _kwargs: + del _kwargs["file_extension"] + else: + _kwargs.update({"file_extension": ext}) + + # Copy any additional global options + if "mlm_client" not in _kwargs and self._mlm_client is not None: + _kwargs["mlm_client"] = self._mlm_client + + if "mlm_model" not in _kwargs and self._mlm_model is not None: + _kwargs["mlm_model"] = self._mlm_model + + # If we hit an error log it and keep trying + # try: + res = converter.convert(local_path, **_kwargs) + # except Exception: + # error_trace = ("\n\n" + traceback.format_exc()).strip() + + if res is not None: + # Normalize the content + res.text_content = "\n".join([line.rstrip() for line in re.split(r"\r?\n", res.text_content)]) + res.text_content = re.sub(r"\n{3,}", "\n\n", res.text_content) + + # Todo + return res + + # If we got this far without success, report any exceptions + if len(error_trace) > 0: + raise FileConversionException( + f"Could not convert '{local_path}' to Markdown. File type was recognized as {extensions}. While converting the file, the following error was encountered:\n\n{error_trace}" + ) + + # Nothing can handle it! + raise UnsupportedFormatException( + f"Could not convert '{local_path}' to Markdown. The formats {extensions} are not supported." + ) + + def _append_ext(self, extensions, ext): + """Append a unique non-None, non-empty extension to a list of extensions.""" + if ext is None: + return + ext = ext.strip() + if ext == "": + return + # if ext not in extensions: + if True: + extensions.append(ext) + + def _guess_ext_magic(self, path): + """Use puremagic (a Python implementation of libmagic) to guess a file's extension based on the first few bytes.""" + # Use puremagic to guess + try: + guesses = puremagic.magic_file(path) + if len(guesses) > 0: + ext = guesses[0].extension.strip() + if len(ext) > 0: + return ext + except FileNotFoundError: + pass + except IsADirectoryError: + pass + except PermissionError: + pass + return None + + def register_page_converter(self, converter: DocumentConverter) -> None: + """Register a page text converter.""" + self._page_converters.insert(0, converter) + import base64 + import io + try: + global page + phtml = page.evaluate("document.documentElement.outerHTML;") + mdconvert = MarkdownConverter() + if page.url == "about:blank": + raise Exception("You cannot convert the content of the blank page. It's meaningless. Make sure you have visited a valid page before converting.") + res = mdconvert.convert_stream(io.StringIO(phtml), file_extension=".html", url=page.url) + + clean_md = f"""# {res.title}\n\n{res.text_content}\n\nIf you have not yet got the answer and want to back to the previous page, please use `visit_url(url={repr(page.url)})`""" + + # 将markdown内容转换为简单的HTML结构 + # 将markdown内容按固定长度分块 + chunk_size = 5000 # 每块大约1000字符 + content = res.text_content + + + chunks = [content[i:i+chunk_size] for i in range(0, len(content), chunk_size)] + + formatted_content = '' + if len(chunks) > 1: + notes = f"The converted markdown text is divided into {len(chunks)} chunks, you can use `page_down()` and `page_up()` to navigate through the text." + else: + notes = "" + for i, chunk in enumerate(chunks): + formatted_content += f''' +
+ {chunk} +
+ ''' + + html_content = f""" + + + {res.title} + + + +
+ {formatted_content} +
+ If you have not yet got the answer and want to back to the previous page, please use `visit_url(url={repr(page.url)})` +
+
+ + + """ + + # 使用base64编码并通过goto显示 + goto( + "data:text/html;base64," + + base64.b64encode(html_content.encode("utf-8")).decode("utf-8") + ) + + # 触发pageshow事件 + page.evaluate(""" + const event = new Event('pageshow', { + bubbles: true, + cancelable: false + }); + window.dispatchEvent(event); + """) + + + # global page + # from playwright.sync_api import sync_playwright + # import io + # with sync_playwright() as p: + # # 启动浏览器 + # url = page.url + # new_browser = p.chromium.launch(headless=True) + # new_page = new_browser.new_page() + # new_page.context.add_cookies([ + # { + # "domain": ".youtube.com", + # "expirationDate": 1718884961, + # "hostOnly": False, + # "httpOnly": False, + # "name": "ST-xuwub9", + # "path": "/", + # "sameSite": "None", + # "secure": False, + # "session": False, + # "storeId": None, + # "value": "session_logininfo=AFmmF2swRAIgf4gadACOuWOcipI1anW-dakEjtidNLkufnOC8uml7EECIDh2YisqWELDBJPTGUysCucJ3I0wjXxYjVHro1LHrdW0%3AQUQ3MjNmd2Jiajl3OWZYRnpFNnZlWWV5ZGJWZ0hpcmp4LVVPU280bk4zOS03Z0ozZG9fOFhWZ0dXaVo3NG1wTEg1b3hGaG10TFBlaFBnTlJfbER5bEp0aFhoNS1OLVhYNFRZT2F6ajgzOFpDbGhlUjZpMWRETlFFRjFfTTRiM0RnNTROSkdmMTFMVjFic1VuZ2trbGp4aktDa0JJUC1BWDh3" + # }, + # ]) + + # # 访问 YouTube 视频 + + # new_page.goto(url, wait_until="networkidle") + + # # 获取页面 HTML + # html = new_page.evaluate("document.documentElement.outerHTML;") + + # # 使用 MarkdownConverter 转换 + # mdconvert = MarkdownConverter() + # res = mdconvert.convert_stream(io.StringIO(html), file_extension=".html", url=url) + + # clean_md = f"""# {res.title}\n\n{res.text_content}""" + + # # 将markdown内容转换为简单的HTML结构 + # html_content = f""" + # + # + # {res.title} + # + # + # + # 
{clean_md}
+ # + # + # """ + + # # 使用base64编码并通过goto显示 + # goto( + # "data:text/html;base64," + + # base64.b64encode(html_content.encode("utf-8")).decode("utf-8") + # ) + + # # 触发pageshow事件 + # page.evaluate(""" + # const event = new Event('pageshow', { + # bubbles: true, + # cancelable: false + # }); + # window.dispatchEvent(event); + # """) + + # # 关闭浏览器 + # new_browser.close() + except Exception as e: + raise Exception(f"Get page markdown error: {str(e)}") +if __name__ == "__main__": + from playwright.sync_api import sync_playwright + import io + with sync_playwright() as p: + # 启动浏览器 + browser = p.chromium.launch(headless=True) + page = browser.new_page() + + # 访问 YouTube 视频 + url = "https://www.researchgate.net/publication/232696279_The_influence_of_social_environment_on_sex_determination_in_harlequin_shrimp_Hymenocera_picta_Decapoda_Gnathophyllidae" + page.goto(url, wait_until="networkidle") + + # 获取页面 HTML + html = page.evaluate("document.documentElement.outerHTML;") + + # 使用 MarkdownConverter 转换 + mdconvert = MarkdownConverter() + res = mdconvert.convert_stream(io.StringIO(html), file_extension=".html", url=url) + + print('标题:', res.title) + print('\n内容:') + print(res.text_content) + + # 关闭浏览器 + browser.close() + # mdconvert = MarkdownConverter() + # res = mdconvert.convert_local("/Users/tangjiabin/Documents/reasoning/autoagent/eval_data/GAIA/2023/validation/2b3ef98c-cc05-450b-a719-711aee40ac65.mp3") + # print('title:', res.title) + # print('content:', res.text_content) \ No newline at end of file diff --git a/autoagent/environment/shutdown_listener.py b/autoagent/environment/shutdown_listener.py new file mode 100644 index 0000000..3aedd26 --- /dev/null +++ b/autoagent/environment/shutdown_listener.py @@ -0,0 +1,65 @@ +""" +This module monitors the app for shutdown signals +""" + +import asyncio +import signal +import threading +import time +from types import FrameType + +from uvicorn.server import HANDLED_SIGNALS + +_should_exit = None + + +def _register_signal_handler(sig: signal.Signals): + original_handler = None + + def handler(sig_: int, frame: FrameType | None): + global _should_exit + _should_exit = True + if original_handler: + original_handler(sig_, frame) # type: ignore[unreachable] + + original_handler = signal.signal(sig, handler) + + +def _register_signal_handlers(): + global _should_exit + if _should_exit is not None: + return + _should_exit = False + + # Check if we're in the main thread of the main interpreter + if threading.current_thread() is threading.main_thread(): + for sig in HANDLED_SIGNALS: + _register_signal_handler(sig) + + +def should_exit() -> bool: + _register_signal_handlers() + return bool(_should_exit) + + +def should_continue() -> bool: + _register_signal_handlers() + return not _should_exit + + +def sleep_if_should_continue(timeout: float): + if timeout <= 1: + time.sleep(timeout) + return + start_time = time.time() + while (time.time() - start_time) < timeout and should_continue(): + time.sleep(1) + + +async def async_sleep_if_should_continue(timeout: float): + if timeout <= 1: + await asyncio.sleep(timeout) + return + start_time = time.time() + while time.time() - start_time < timeout and should_continue(): + await asyncio.sleep(1) diff --git a/autoagent/environment/tcp_server.py b/autoagent/environment/tcp_server.py new file mode 100644 index 0000000..9ad41c8 --- /dev/null +++ b/autoagent/environment/tcp_server.py @@ -0,0 +1,80 @@ +import socket +import subprocess +import json +import argparse + +parser = argparse.ArgumentParser() +parser.add_argument("--workplace", type=str, default=None) +parser.add_argument("--conda_path", type=str, default=None) +parser.add_argument("--port", type=int, default=None) +args = parser.parse_args() + +if __name__ == "__main__": + assert args.workplace is not None, "Workplace is not specified" + assert args.conda_path is not None, "Conda path is not specified" + assert args.port is not None, "Port is not specified" + server = socket.socket(socket.AF_INET, socket.SOCK_STREAM) + server.bind(("0.0.0.0", args.port)) + server.listen(1) + + print(f"Listening on port {args.port}...") + def receive_all(conn, buffer_size=4096): + data = b"" + while True: + part = conn.recv(buffer_size) + data += part + if len(part) < buffer_size: + # 如果接收的数据小于缓冲区大小，可能已经接收完毕 + break + return data.decode() + + while True: + conn, addr = server.accept() + print(f"Connection from {addr}") + while True: + # command = conn.recv(1024).decode() + command = receive_all(conn) + if not command: + break + + # Execute the command + try: + modified_command = f"/bin/bash -c 'source {args.conda_path}/etc/profile.d/conda.sh && conda activate autogpt && cd /{args.workplace} && {command}'" + process = subprocess.Popen(modified_command, shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True) + output = '' + while True: + line = process.stdout.readline() + if not line and process.poll() is not None: + break + output += line + # 立即发送每一行输出 + chunk_response = { + "type": "chunk", + "data": line + } + conn.send(json.dumps(chunk_response).encode() + b"\n") # 添加换行符作为分隔符 + + # 发送最终的完整响应 + final_response = { + "type": "final", + "status": process.poll(), + "result": output + } + conn.send(json.dumps(final_response).encode() + b"\n") + except Exception as e: + error_response = { + "type": "final", + "status": -1, + "result": f"Error running command: {str(e)}" + } + conn.send(json.dumps(error_response).encode() + b"\n") + + # Create a JSON response + # response = { + # "status": exit_code, + # "result": output + # } + + # # Send the JSON response + # conn.send(json.dumps(response).encode()) + conn.close() \ No newline at end of file diff --git a/autoagent/environment/tenacity_stop.py b/autoagent/environment/tenacity_stop.py new file mode 100644 index 0000000..c1edabd --- /dev/null +++ b/autoagent/environment/tenacity_stop.py @@ -0,0 +1,11 @@ +from tenacity import RetryCallState +from tenacity.stop import stop_base + +from .shutdown_listener import should_exit + + +class stop_if_should_exit(stop_base): + """Stop if the should_exit flag is set.""" + + def __call__(self, retry_state: 'RetryCallState') -> bool: + return should_exit() diff --git a/autoagent/environment/utils.py b/autoagent/environment/utils.py new file mode 100644 index 0000000..bbf7c33 --- /dev/null +++ b/autoagent/environment/utils.py @@ -0,0 +1,16 @@ +from autoagent.util import run_command_in_container +from .docker_env import DockerEnv +from autoagent.io_utils import print_stream +def setup_metachain(workplace_name: str, env: DockerEnv): + cmd = "pip list | grep autoagent" + response = env.run_command(cmd, print_stream) + if response['status'] == 0: + print("AutoAgent is already installed.") + return + cmd = f"cd /{workplace_name}/AutoAgent && pip install -e ." + response = env.run_command(cmd, print_stream) + if response['status'] == 0: + print("AutoAgent is installed.") + return + else: + raise Exception(f"Failed to install autoagent. {response['result']}") diff --git a/autoagent/flow/__init__.py b/autoagent/flow/__init__.py new file mode 100644 index 0000000..dd74af2 --- /dev/null +++ b/autoagent/flow/__init__.py @@ -0,0 +1,4 @@ +from .core import EventEngineCls +from .types import EventInput, ReturnBehavior + +default_drive = EventEngineCls() diff --git a/autoagent/flow/broker.py b/autoagent/flow/broker.py new file mode 100644 index 0000000..7abf1d5 --- /dev/null +++ b/autoagent/flow/broker.py @@ -0,0 +1,11 @@ +from typing import Any +from .types import BaseEvent, EventInput, Task, GroupEventReturns +from .utils import generate_uuid + + +class BaseBroker: + async def append(self, event: BaseEvent, event_input: EventInput) -> Task: + raise NotImplementedError() + + async def callback_after_run_done(self) -> tuple[BaseEvent, Any]: + raise NotImplementedError() diff --git a/autoagent/flow/core.py b/autoagent/flow/core.py new file mode 100644 index 0000000..1382885 --- /dev/null +++ b/autoagent/flow/core.py @@ -0,0 +1,175 @@ +import inspect +import asyncio +from typing import Callable, Optional, Union, Any, Tuple, Literal +from .types import ( + BaseEvent, + EventFunction, + EventGroup, + EventInput, + _SpecialEventReturn, + ReturnBehavior, + InvokeInterCache, +) +from .broker import BaseBroker +from .utils import logger, string_to_md5_hash, generate_uuid + + +class EventEngineCls: + def __init__(self, name="default", broker: Optional[BaseBroker] = None): + self.name = name + self.broker = broker or BaseBroker() + self.__event_maps: dict[str, BaseEvent] = {} + self.__max_group_size = 0 + + def reset(self): + self.__event_maps = {} + + def get_event_from_id(self, event_id: str) -> Optional[BaseEvent]: + return self.__event_maps.get(event_id) + + def make_event(self, func: Union[EventFunction, BaseEvent]) -> BaseEvent: + if isinstance(func, BaseEvent): + self.__event_maps[func.id] = func + return func + assert inspect.iscoroutinefunction( + func + ), "Event function must be a coroutine function" + event = BaseEvent(func) + self.__event_maps[event.id] = event + return event + + def listen_group( + self, + group_markers: list[BaseEvent], + group_name: Optional[str] = None, + retrigger_type: Literal["all", "any"] = "all", + ) -> Callable[[BaseEvent], BaseEvent]: + assert all( + [isinstance(m, BaseEvent) for m in group_markers] + ), "group_markers must be a list of BaseEvent" + assert all( + [m.id in self.__event_maps for m in group_markers] + ), f"group_markers must be registered in the same event engine, current event engine is {self.name}" + group_markers_in_dict = {event.id: event for event in group_markers} + + def decorator(func: BaseEvent) -> BaseEvent: + if not isinstance(func, BaseEvent): + func = self.make_event(func) + assert ( + func.id in self.__event_maps + ), f"Event function must be registered in the same event engine, current event engine is {self.name}" + this_group_name = group_name or f"{len(func.parent_groups)}" + this_group_hash = string_to_md5_hash(":".join(group_markers_in_dict.keys())) + new_group = EventGroup( + this_group_name, + this_group_hash, + group_markers_in_dict, + retrigger_type=retrigger_type, + ) + self.__max_group_size = max( + self.__max_group_size, len(group_markers_in_dict) + ) + if new_group.hash() in func.parent_groups: + logger.warning(f"Group {group_markers} already listened by {func}") + return func + func.parent_groups[new_group.hash()] = new_group + return func + + return decorator + + def goto(self, group_markers: list[BaseEvent], *args): + raise NotImplementedError() + + async def invoke_event( + self, + event: BaseEvent, + event_input: Optional[EventInput] = None, + global_ctx: Any = None, + max_async_events: Optional[int] = None, + ) -> dict[str, Any]: + this_run_ctx: dict[str, InvokeInterCache] = {} + queue: list[Tuple[str, EventInput]] = [(event.id, event_input)] + + async def run_event(current_event_id: str, current_event_input: Any): + current_event = self.get_event_from_id(current_event_id) + assert current_event is not None, f"Event {current_event_id} not found" + result = await current_event.solo_run(current_event_input, global_ctx) + this_run_ctx[current_event.id] = { + "result": result, + "already_sent_to_event_group": set(), + } + if isinstance(result, _SpecialEventReturn): + if result.behavior == ReturnBehavior.GOTO: + group_markers, any_return = result.returns + for group_marker in group_markers: + this_group_returns = {current_event.id: any_return} + build_input_goto = EventInput( + group_name="$goto", + results=this_group_returns, + behavior=ReturnBehavior.GOTO, + ) + queue.append((group_marker.id, build_input_goto)) + elif result.behavior == ReturnBehavior.ABORT: + return + else: + # dispath to events who listen + for cand_event in self.__event_maps.values(): + cand_event_parents = cand_event.parent_groups + for group_hash, group in cand_event_parents.items(): + if_current_event_trigger = current_event.id in group.events + if_ctx_cover = all( + [event_id in this_run_ctx for event_id in group.events] + ) + event_group_id = f"{cand_event.id}:{group_hash}" + if if_current_event_trigger and if_ctx_cover: + if ( + any( + [ + event_group_id + in this_run_ctx[event_id][ + "already_sent_to_event_group" + ] + for event_id in group.events + ] + ) + and group.retrigger_type == "all" + ): + # some events already dispatched to this event and group, skip + logger.debug(f"Skip {cand_event} for {current_event}") + continue + this_group_returns = { + event_id: this_run_ctx[event_id]["result"] + for event_id in group.events + } + for event_id in group.events: + this_run_ctx[event_id][ + "already_sent_to_event_group" + ].add(event_group_id) + build_input = EventInput( + group_name=group.name, results=this_group_returns + ) + queue.append((cand_event.id, build_input)) + + tasks = set() + try: + while len(queue) or len(tasks): + this_batch_events = ( + queue[:max_async_events] if max_async_events else queue + ) + queue = queue[max_async_events:] if max_async_events else [] + new_tasks = { + asyncio.create_task(run_event(*run_event_input)) + for run_event_input in this_batch_events + } + tasks.update(new_tasks) + done, tasks = await asyncio.wait( + tasks, return_when=asyncio.FIRST_COMPLETED + ) + for task in done: + await task # Handle any exceptions + except asyncio.CancelledError: + for task in tasks: + task.cancel() + await asyncio.gather(*tasks, return_exceptions=True) + raise + return this_run_ctx diff --git a/autoagent/flow/dynamic.py b/autoagent/flow/dynamic.py new file mode 100644 index 0000000..048c9ed --- /dev/null +++ b/autoagent/flow/dynamic.py @@ -0,0 +1,18 @@ +from typing import Any +from .types import ( + BaseEvent, + _SpecialEventReturn, + ReturnBehavior, +) + + +def goto_events( + group_markers: list[BaseEvent], any_return: Any = None +) -> _SpecialEventReturn: + return _SpecialEventReturn( + behavior=ReturnBehavior.GOTO, returns=(group_markers, any_return) + ) + + +def abort_this(): + return _SpecialEventReturn(behavior=ReturnBehavior.ABORT, returns=None) diff --git a/autoagent/flow/types.py b/autoagent/flow/types.py new file mode 100644 index 0000000..3dd52f6 --- /dev/null +++ b/autoagent/flow/types.py @@ -0,0 +1,146 @@ +from copy import copy +from enum import Enum +from dataclasses import dataclass, field +from datetime import datetime +from typing import Any, Awaitable, Optional, Union, Callable, TypedDict, Literal + +from .utils import ( + string_to_md5_hash, + generate_uuid, + function_or_method_to_string, + function_or_method_to_repr, +) + + +class ReturnBehavior(Enum): + DISPATCH = "dispatch" + GOTO = "goto" + ABORT = "abort" + INPUT = "input" + + +class TaskStatus(Enum): + RUNNING = "running" + SUCCESS = "success" + FAILURE = "failure" + PENDING = "pending" + + +class InvokeInterCache(TypedDict): + result: Any + already_sent_to_event_group: set[str] + + +GroupEventReturns = dict[str, Any] + + +@dataclass +class EventGroupInput: + group_name: str + results: GroupEventReturns + behavior: ReturnBehavior = ReturnBehavior.DISPATCH + + +@dataclass +class EventInput(EventGroupInput): + task_id: str = field(default_factory=generate_uuid) + + @classmethod + def from_input(cls: "EventInput", input_data: dict[str, Any]) -> "EventInput": + return cls( + group_name="user_input", results=input_data, behavior=ReturnBehavior.INPUT + ) + + +@dataclass +class _SpecialEventReturn: + behavior: ReturnBehavior + returns: Any + + def __post_init__(self): + if not isinstance(self.behavior, ReturnBehavior): + raise TypeError( + f"behavior must be a ReturnBehavior, not {type(self.behavior)}" + ) + + +# (group_event_results, global ctx set by user) -> result +EventFunction = Callable[ + [Optional[EventInput], Optional[Any]], Awaitable[Union[Any, _SpecialEventReturn]] +] + + +@dataclass +class EventGroup: + name: str + events_hash: str + events: dict[str, "BaseEvent"] + retrigger_type: Literal["all", "any"] = "all" + + def hash(self) -> str: + return self.events_hash + + +class BaseEvent: + parent_groups: dict[str, EventGroup] + func_inst: EventFunction + id: str + repr_name: str + + def __init__( + self, + func_inst: EventFunction, + parent_groups: Optional[dict[str, EventGroup]] = None, + ): + self.parent_groups = parent_groups or {} + self.func_inst = func_inst + self.id = string_to_md5_hash(function_or_method_to_string(self.func_inst)) + self.repr_name = function_or_method_to_repr(self.func_inst) + self.meta = {"func_body": function_or_method_to_string(self.func_inst)} + + def debug_string(self, exclude_events: Optional[set[str]] = None) -> str: + exclude_events = exclude_events or set([self.id]) + parents_str = format_parents(self.parent_groups, exclude_events=exclude_events) + return f"{self.repr_name}\n{parents_str}" + + def __repr__(self) -> str: + return f"Node(source={self.repr_name})" + + async def solo_run( + self, event_input: EventInput, global_ctx: Any = None + ) -> Awaitable[Any]: + return await self.func_inst(event_input, global_ctx) + + +@dataclass +class Task: + task_id: str + status: TaskStatus = TaskStatus.PENDING + created_at: datetime = field(default_factory=datetime.now) + upated_at: datetime = field(default_factory=datetime.now) + + +def format_parents(parents: dict[str, EventGroup], exclude_events: set[str], indent=""): + # Below code is ugly + # But it works and only for debug display + result = [] + for i, parent_group in enumerate(parents.values()): + is_last_group = i == len(parents) - 1 + group_prefix = "└─ " if is_last_group else "├─ " + result.append(indent + group_prefix + f"<{parent_group.name}>") + for j, parent in enumerate(parent_group.events.values()): + root_events = copy(exclude_events) + is_last = j == len(parent_group.events) - 1 + child_indent = indent + (" " if is_last_group else "│ ") + inter_indent = " " if is_last else "│ " + prefix = "└─ " if is_last else "├─ " + if parent.id in root_events: + result.append(f"{child_indent}{prefix}{parent.repr_name} ") + continue + root_events.add(parent.id) + parent_debug = parent.debug_string(exclude_events=root_events).split("\n") + parent_debug = [p for p in parent_debug if p.strip()] + result.append(f"{child_indent}{prefix}{parent.repr_name}") + for line in parent_debug[1:]: + result.append(f"{child_indent}{inter_indent}{line}") + return "\n".join(result) diff --git a/autoagent/flow/utils.py b/autoagent/flow/utils.py new file mode 100644 index 0000000..86cf4a7 --- /dev/null +++ b/autoagent/flow/utils.py @@ -0,0 +1,48 @@ +import uuid +import logging +import asyncio +import inspect +import hashlib +from typing import Callable + +logger = logging.getLogger("drive-flow") + + +def generate_uuid() -> str: + return str(uuid.uuid4()) + + +def function_or_method_to_repr(func_or_method: Callable) -> str: + is_method = inspect.ismethod(func_or_method) + is_function = inspect.isfunction(func_or_method) + if not is_method and not is_function: + raise ValueError("Input must be a function or method") + module = func_or_method.__module__ + name = func_or_method.__name__ + line_number = inspect.getsourcelines(func_or_method)[1] + + if is_method: + class_name = func_or_method.__self__.__class__.__name__ + return f"{module}.l_{line_number}.{class_name}.{name}".strip() + else: + return f"{module}.l_{line_number}.{name}".strip() + + +def function_or_method_to_string(func_or_method: Callable) -> str: + is_method = inspect.ismethod(func_or_method) + is_function = inspect.isfunction(func_or_method) + if not is_method and not is_function: + raise ValueError("Input must be a function or method") + module = func_or_method.__module__ + source = inspect.getsource(func_or_method) + line_number = inspect.getsourcelines(func_or_method)[1] + + if is_method: + class_name = func_or_method.__self__.__class__.__name__ + return f"{module}.l_{line_number}.{class_name}\n{source}".strip() + else: + return f"{module}.l_{line_number}\n{source}".strip() + + +def string_to_md5_hash(string: str) -> str: + return hashlib.md5(string.encode()).hexdigest() diff --git a/autoagent/fn_call_converter.py b/autoagent/fn_call_converter.py new file mode 100644 index 0000000..315c9c3 --- /dev/null +++ b/autoagent/fn_call_converter.py @@ -0,0 +1,848 @@ +"""Convert function calling messages to non-function calling messages and vice versa. + +This will inject prompts so that models that doesn't support function calling +can still be used with function calling agents. + +We follow format from: https://docs.litellm.ai/docs/completion/function_call +""" + +import copy +import json +import re +from typing import Iterable + +from litellm import ChatCompletionToolParam + +class FunctionCallConversionError(Exception): + """Exception raised when FunctionCallingConverter failed to convert a non-function call message to a function call message. + + This typically happens when there's a malformed message (e.g., missing tags). But not due to LLM output. + """ + + def __init__(self, message): + super().__init__(message) + +class FunctionCallValidationError(Exception): + """Exception raised when FunctionCallingConverter failed to validate a function call message. + + This typically happens when the LLM outputs unrecognized function call / parameter names / values. + """ + + def __init__(self, message): + super().__init__(message) + +# Inspired by: https://docs.together.ai/docs/llama-3-function-calling#function-calling-w-llama-31-70b +SYSTEM_PROMPT_SUFFIX_TEMPLATE = """ +You have access to the following functions: + +{description} + +If you choose to call a function ONLY reply in the following format with NO suffix: + + +value_1 + +This is the value for the second parameter +that can span +multiple lines + + + + +Reminder: +- Function calls MUST follow the specified format, start with +- Required parameters MUST be specified +- Only call one function at a time +- You may provide optional reasoning for your function call in natural language BEFORE the function call, but NOT after. +- If there is no function call available, answer the question like normal with your current knowledge and do not tell the user about function calls +""" + +STOP_WORDS = [' + +pwd && ls + + + +USER: EXECUTION RESULT of [execute_bash]: +/workspace +openhands@runtime:~/workspace$ + +ASSISTANT: There is no `app.py` file in the current directory. Let me create a Python file `app.py`: + +create +/workspace/app.py + +from flask import Flask +app = Flask(__name__) + +@app.route('/') +def index(): + numbers = list(range(1, 11)) + return str(numbers) + +if __name__ == '__main__': + app.run(port=5000) + + + +USER: EXECUTION RESULT of [str_replace_editor]: +File created successfully at: /workspace/app.py + +ASSISTANT: I have created a Python file `app.py` that will display a list of numbers from 1 to 10 when you run it. Let me run the Python file for you: + + +python3 app.py > server.log 2>&1 & + + + +USER: EXECUTION RESULT of [execute_bash]: +[1] 121 +[1]+ Exit 1 python3 app.py > server.log 2>&1 + +ASSISTANT: Looks like the server is running with PID 121 then crashed. Let me check the server log: + + +cat server.log + + + +USER: EXECUTION RESULT of [execute_bash]: +Traceback (most recent call last): + File "/workspace/app.py", line 2, in + from flask import Flask +ModuleNotFoundError: No module named 'flask' + +ASSISTANT: Looks like the server crashed because the `flask` module is not installed. Let me install the `flask` module for you: + + +pip3 install flask + + + +USER: EXECUTION RESULT of [execute_bash]: +Defaulting to user installation because normal site-packages is not writeable +Collecting flask + Using cached flask-3.0.3-py3-none-any.whl (101 kB) +Collecting blinker>=1.6.2 + Using cached blinker-1.7.0-py3-none-any.whl (13 kB) +Collecting Werkzeug>=3.0.0 + Using cached werkzeug-3.0.2-py3-none-any.whl (226 kB) +Collecting click>=8.1.3 + Using cached click-8.1.7-py3-none-any.whl (97 kB) +Collecting itsdangerous>=2.1.2 + Using cached itsdangerous-2.2.0-py3-none-any.whl (16 kB) +Requirement already satisfied: Jinja2>=3.1.2 in /home/openhands/.local/lib/python3.10/site-packages (from flask) (3.1.3) +Requirement already satisfied: MarkupSafe>=2.0 in /home/openhands/.local/lib/python3.10/site-packages (from Jinja2>=3.1.2->flask) (2.1.5) +Installing collected packages: Werkzeug, itsdangerous, click, blinker, flask +Successfully installed Werkzeug-3.0.2 blinker-1.7.0 click-8.1.7 flask-3.0.3 itsdangerous-2.2.0 + +ASSISTANT: The `flask` module is installed successfully. Let me run the Python file again: + + +python3 app.py > server.log 2>&1 & + + + +USER: EXECUTION RESULT of [execute_bash]: +[1] 124 + +ASSISTANT: +Let me check the server log again: + + +cat server.log + + + +USER: EXECUTION RESULT of [execute_bash]: +* Serving Flask app 'app' + * Debug mode: off +WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead. + * Running on http://127.0.0.1:5000 +Press CTRL+C to quit + +ASSISTANT: +The server is running on port 5000 with PID 124. You can access the list of numbers by visiting http://127.0.0.1:5000. If you have any further questions, feel free to ask! + +USER: Now kill the server, make it display the numbers in a table format. + +ASSISTANT: +Sure! Let me stop the server first: + + +kill 124 + + + +USER: EXECUTION RESULT of [execute_bash]: +[1]+ Terminated python3 app.py > server.log 2>&1 + +ASSISTANT: +Now let me display the numbers in a table format: + +str_replace +/workspace/app.py +return str(numbers) +return '' + ''.join([f'' for i in numbers]) + '
{i}
' + + +USER: EXECUTION RESULT of [str_replace_editor]: +The file /workspace/app.py has been edited. Here's the result of running `cat -n` on a snippet of /workspace/app.py: + 3 + 4 @app.route('/') + 5 def index(): + 6 numbers = list(range(1, 11)) + 7 return '' + ''.join([f'' for i in numbers]) + '
{i}
' + 8 + 9 if __name__ == '__main__': + 10 app.run(port=5000) +Review the changes and make sure they are as expected. Edit the file again if necessary. + +ASSISTANT: +Running the updated file: + + + +python3 app.py > server.log 2>&1 & + + + +USER: EXECUTION RESULT of [execute_bash]: +[1] 126 + +ASSISTANT: +The server is running on port 5000 with PID 126. You can access the list of numbers in a table format by visiting http://127.0.0.1:5000. Let me know if you have any further requests! + + + +--------------------- END OF EXAMPLE --------------------- + +Do NOT assume the environment is the same as in the example above. + +--------------------- NEW TASK DESCRIPTION --------------------- +""".lstrip() + +IN_CONTEXT_LEARNING_EXAMPLE_SUFFIX = """ +--------------------- END OF NEW TASK DESCRIPTION --------------------- + +PLEASE follow the format strictly! PLEASE EMIT ONE AND ONLY ONE FUNCTION CALL PER MESSAGE. +""".lstrip() + +# Regex patterns for function call parsing +FN_REGEX_PATTERN = r']+)>\n(.*?)' +FN_PARAM_REGEX_PATTERN = r']+)>(.*?)' + +# Add new regex pattern for tool execution results +TOOL_RESULT_REGEX_PATTERN = r'EXECUTION RESULT of \[(.*?)\]:\n(.*)' + + +def convert_tool_call_to_string(tool_call: dict) -> str: + """Convert tool call to content in string format.""" + if 'function' not in tool_call: + raise FunctionCallConversionError("Tool call must contain 'function' key.") + if 'id' not in tool_call: + raise FunctionCallConversionError("Tool call must contain 'id' key.") + if 'type' not in tool_call: + raise FunctionCallConversionError("Tool call must contain 'type' key.") + if tool_call['type'] != 'function': + raise FunctionCallConversionError("Tool call type must be 'function'.") + + ret = f"\n" + try: + args = json.loads(tool_call['function']['arguments']) + except json.JSONDecodeError as e: + raise FunctionCallConversionError( + f"Failed to parse arguments as JSON. Arguments: {tool_call['function']['arguments']}" + ) from e + for param_name, param_value in args.items(): + is_multiline = isinstance(param_value, str) and '\n' in param_value + ret += f'' + if is_multiline: + ret += '\n' + ret += f'{param_value}' + if is_multiline: + ret += '\n' + ret += '\n' + ret += '' + return ret + + +def convert_tools_to_description(tools: list[dict]) -> str: + ret = '' + for i, tool in enumerate(tools): + assert tool['type'] == 'function' + fn = tool['function'] + if i > 0: + ret += '\n' + ret += f"---- BEGIN FUNCTION #{i+1}: {fn['name']} ----\n" + ret += f"Description: {fn['description']}\n" + + if 'parameters' in fn: + ret += 'Parameters:\n' + properties = fn['parameters'].get('properties', {}) + required_params = set(fn['parameters'].get('required', [])) + + for j, (param_name, param_info) in enumerate(properties.items()): + # Indicate required/optional in parentheses with type + is_required = param_name in required_params + param_status = 'required' if is_required else 'optional' + param_type = param_info.get('type', 'string') + + # Get parameter description + desc = param_info.get('description', 'No description provided') + + # Handle enum values if present + if 'enum' in param_info: + enum_values = ', '.join(f'`{v}`' for v in param_info['enum']) + desc += f'\nAllowed values: [{enum_values}]' + + ret += ( + f' ({j+1}) {param_name} ({param_type}, {param_status}): {desc}\n' + ) + else: + ret += 'No parameters are required for this function.\n' + + ret += f'---- END FUNCTION #{i+1} ----\n' + return ret + + +def convert_fncall_messages_to_non_fncall_messages( + messages: list[dict], + tools: list[ChatCompletionToolParam], + add_in_context_learning_example: bool = True, +) -> list[dict]: + """Convert function calling messages to non-function calling messages.""" + messages = copy.deepcopy(messages) + + formatted_tools = convert_tools_to_description(tools) + system_prompt_suffix = SYSTEM_PROMPT_SUFFIX_TEMPLATE.format( + description=formatted_tools + ) + + converted_messages = [] + first_user_message_encountered = False + for message in messages: + role = message['role'] + content = message['content'] + + # 1. SYSTEM MESSAGES + # append system prompt suffix to content + if role == 'system': + if isinstance(content, str): + content += system_prompt_suffix + elif isinstance(content, list): + if content and content[-1]['type'] == 'text': + content[-1]['text'] += system_prompt_suffix + else: + content.append({'type': 'text', 'text': system_prompt_suffix}) + else: + raise FunctionCallConversionError( + f'Unexpected content type {type(content)}. Expected str or list. Content: {content}' + ) + converted_messages.append({'role': 'system', 'content': content}) + + # 2. USER MESSAGES (no change) + elif role == 'user': + # Add in-context learning example for the first user message + if not first_user_message_encountered and add_in_context_learning_example: + first_user_message_encountered = True + # Check tools + if not ( + tools + and len(tools) > 0 + and any( + ( + tool['type'] == 'function' + and tool['function']['name'] == 'execute_bash' + and 'command' + in tool['function']['parameters']['properties'] + ) + for tool in tools + ) + and any( + ( + tool['type'] == 'function' + and tool['function']['name'] == 'str_replace_editor' + and 'path' in tool['function']['parameters']['properties'] + and 'file_text' + in tool['function']['parameters']['properties'] + and 'old_str' + in tool['function']['parameters']['properties'] + and 'new_str' + in tool['function']['parameters']['properties'] + ) + for tool in tools + ) + ): + raise FunctionCallConversionError( + 'The currently provided tool set are NOT compatible with the in-context learning example for FnCall to Non-FnCall conversion. ' + 'Please update your tool set OR the in-context learning example in openhands/llm/fn_call_converter.py' + ) + + # add in-context learning example + if isinstance(content, str): + content = ( + IN_CONTEXT_LEARNING_EXAMPLE_PREFIX + + content + + IN_CONTEXT_LEARNING_EXAMPLE_SUFFIX + ) + elif isinstance(content, list): + if content and content[0]['type'] == 'text': + content[0]['text'] = ( + IN_CONTEXT_LEARNING_EXAMPLE_PREFIX + + content[0]['text'] + + IN_CONTEXT_LEARNING_EXAMPLE_SUFFIX + ) + else: + content = ( + [ + { + 'type': 'text', + 'text': IN_CONTEXT_LEARNING_EXAMPLE_PREFIX, + } + ] + + content + + [ + { + 'type': 'text', + 'text': IN_CONTEXT_LEARNING_EXAMPLE_SUFFIX, + } + ] + ) + else: + raise FunctionCallConversionError( + f'Unexpected content type {type(content)}. Expected str or list. Content: {content}' + ) + converted_messages.append( + { + 'role': 'user', + 'content': content, + } + ) + + # 3. ASSISTANT MESSAGES + # - 3.1 no change if no function call + # - 3.2 change if function call + elif role == 'assistant': + if 'tool_calls' in message and message['tool_calls'] is not None: + if len(message['tool_calls']) != 1: + raise FunctionCallConversionError( + f'Expected exactly one tool call in the message. More than one tool call is not supported. But got {len(message["tool_calls"])} tool calls. Content: {content}' + ) + try: + tool_content = convert_tool_call_to_string(message['tool_calls'][0]) + except FunctionCallConversionError as e: + raise FunctionCallConversionError( + f'Failed to convert tool call to string.\nCurrent tool call: {message["tool_calls"][0]}.\nRaw messages: {json.dumps(messages, indent=2)}' + ) from e + if isinstance(content, str): + content += '\n\n' + tool_content + content = content.lstrip() + elif isinstance(content, list): + if content and content[-1]['type'] == 'text': + content[-1]['text'] += '\n\n' + tool_content + content[-1]['text'] = content[-1]['text'].lstrip() + else: + content.append({'type': 'text', 'text': tool_content}) + else: + raise FunctionCallConversionError( + f'Unexpected content type {type(content)}. Expected str or list. Content: {content}' + ) + converted_messages.append({'role': 'assistant', 'content': content}) + + # 4. TOOL MESSAGES (tool outputs) + elif role == 'tool': + # Convert tool result as user message + tool_name = message.get('name', 'function') + prefix = f'EXECUTION RESULT of [{tool_name}]:\n' + # and omit "tool_call_id" AND "name" + if isinstance(content, str): + content = prefix + content + elif isinstance(content, list): + if content and content[-1]['type'] == 'text': + content[-1]['text'] = prefix + content[-1]['text'] + else: + content = [{'type': 'text', 'text': prefix}] + content + else: + raise FunctionCallConversionError( + f'Unexpected content type {type(content)}. Expected str or list. Content: {content}' + ) + converted_messages.append({'role': 'user', 'content': content}) + else: + raise FunctionCallConversionError( + f'Unexpected role {role}. Expected system, user, assistant or tool.' + ) + return converted_messages + + +def _extract_and_validate_params( + matching_tool: dict, param_matches: Iterable[re.Match], fn_name: str +) -> dict: + params = {} + # Parse and validate parameters + required_params = set() + if 'parameters' in matching_tool and 'required' in matching_tool['parameters']: + required_params = set(matching_tool['parameters'].get('required', [])) + + allowed_params = set() + if 'parameters' in matching_tool and 'properties' in matching_tool['parameters']: + allowed_params = set(matching_tool['parameters']['properties'].keys()) + + param_name_to_type = {} + if 'parameters' in matching_tool and 'properties' in matching_tool['parameters']: + param_name_to_type = { + name: val.get('type', 'string') + for name, val in matching_tool['parameters']['properties'].items() + } + + # Collect parameters + found_params = set() + for param_match in param_matches: + param_name = param_match.group(1) + param_value = param_match.group(2).strip() + + # Validate parameter is allowed + if allowed_params and param_name not in allowed_params: + raise FunctionCallValidationError( + f"Parameter '{param_name}' is not allowed for function '{fn_name}'. " + f'Allowed parameters: {allowed_params}' + ) + + # Validate and convert parameter type + # supported: string, integer, array + if param_name in param_name_to_type: + if param_name_to_type[param_name] == 'integer': + try: + param_value = int(param_value) + except ValueError: + raise FunctionCallValidationError( + f"Parameter '{param_name}' is expected to be an integer." + ) + elif param_name_to_type[param_name] == 'array': + try: + param_value = json.loads(param_value) + except json.JSONDecodeError: + raise FunctionCallValidationError( + f"Parameter '{param_name}' is expected to be an array." + ) + else: + # string + pass + + # Enum check + if 'enum' in matching_tool['parameters']['properties'][param_name]: + if ( + param_value + not in matching_tool['parameters']['properties'][param_name]['enum'] + ): + raise FunctionCallValidationError( + f"Parameter '{param_name}' is expected to be one of {matching_tool['parameters']['properties'][param_name]['enum']}." + ) + + params[param_name] = param_value + found_params.add(param_name) + + # Check all required parameters are present + missing_params = required_params - found_params + if missing_params: + raise FunctionCallValidationError( + f"Missing required parameters for function '{fn_name}': {missing_params}" + ) + return params + + +def _fix_stopword(content: str) -> str: + """Fix the issue when some LLM would NOT return the stopword.""" + if '' + else: + content = content + '\n' + return content + + +def convert_non_fncall_messages_to_fncall_messages( + messages: list[dict], + tools: list[ChatCompletionToolParam], +) -> list[dict]: + """Convert non-function calling messages back to function calling messages.""" + messages = copy.deepcopy(messages) + formatted_tools = convert_tools_to_description(tools) + system_prompt_suffix = SYSTEM_PROMPT_SUFFIX_TEMPLATE.format( + description=formatted_tools + ) + + converted_messages = [] + tool_call_counter = 1 # Counter for tool calls + + first_user_message_encountered = False + for message in messages: + role, content = message['role'], message['content'] + content = content or '' # handle cases where content is None + # For system messages, remove the added suffix + if role == 'system': + if isinstance(content, str): + # Remove the suffix if present + content = content.split(system_prompt_suffix)[0] + elif isinstance(content, list): + if content and content[-1]['type'] == 'text': + # Remove the suffix from the last text item + content[-1]['text'] = content[-1]['text'].split( + system_prompt_suffix + )[0] + converted_messages.append({'role': 'system', 'content': content}) + # Skip user messages (no conversion needed) + elif role == 'user': + # Check & replace in-context learning example + if not first_user_message_encountered: + first_user_message_encountered = True + if isinstance(content, str): + content = content.replace(IN_CONTEXT_LEARNING_EXAMPLE_PREFIX, '') + content = content.replace(IN_CONTEXT_LEARNING_EXAMPLE_SUFFIX, '') + elif isinstance(content, list): + for item in content: + if item['type'] == 'text': + item['text'] = item['text'].replace( + IN_CONTEXT_LEARNING_EXAMPLE_PREFIX, '' + ) + item['text'] = item['text'].replace( + IN_CONTEXT_LEARNING_EXAMPLE_SUFFIX, '' + ) + else: + raise FunctionCallConversionError( + f'Unexpected content type {type(content)}. Expected str or list. Content: {content}' + ) + + # Check for tool execution result pattern + if isinstance(content, str): + tool_result_match = re.search( + TOOL_RESULT_REGEX_PATTERN, content, re.DOTALL + ) + elif isinstance(content, list): + tool_result_match = next( + ( + _match + for item in content + if item.get('type') == 'text' + and ( + _match := re.search( + TOOL_RESULT_REGEX_PATTERN, item['text'], re.DOTALL + ) + ) + ), + None, + ) + else: + raise FunctionCallConversionError( + f'Unexpected content type {type(content)}. Expected str or list. Content: {content}' + ) + + if tool_result_match: + if not ( + isinstance(content, str) + or ( + isinstance(content, list) + and len(content) == 1 + and content[0].get('type') == 'text' + ) + ): + raise FunctionCallConversionError( + f'Expected str or list with one text item when tool result is present in the message. Content: {content}' + ) + tool_name = tool_result_match.group(1) + tool_result = tool_result_match.group(2).strip() + + # Convert to tool message format + converted_messages.append( + { + 'role': 'tool', + 'name': tool_name, + 'content': [{'type': 'text', 'text': tool_result}] + if isinstance(content, list) + else tool_result, + 'tool_call_id': f'toolu_{tool_call_counter-1:02d}', # Use last generated ID + } + ) + else: + converted_messages.append({'role': 'user', 'content': content}) + + # Handle assistant messages + elif role == 'assistant': + if isinstance(content, str): + content = _fix_stopword(content) + fn_match = re.search(FN_REGEX_PATTERN, content, re.DOTALL) + elif isinstance(content, list): + if content and content[-1]['type'] == 'text': + content[-1]['text'] = _fix_stopword(content[-1]['text']) + fn_match = re.search( + FN_REGEX_PATTERN, content[-1]['text'], re.DOTALL + ) + else: + fn_match = None + fn_match_exists = any( + item.get('type') == 'text' + and re.search(FN_REGEX_PATTERN, item['text'], re.DOTALL) + for item in content + ) + if fn_match_exists and not fn_match: + raise FunctionCallConversionError( + f'Expecting function call in the LAST index of content list. But got content={content}' + ) + else: + raise FunctionCallConversionError( + f'Unexpected content type {type(content)}. Expected str or list. Content: {content}' + ) + + if fn_match: + fn_name = fn_match.group(1) + fn_body = fn_match.group(2) + matching_tool = next( + ( + tool['function'] + for tool in tools + if tool['type'] == 'function' + and tool['function']['name'] == fn_name + ), + None, + ) + # Validate function exists in tools + if not matching_tool: + raise FunctionCallValidationError( + f"Function '{fn_name}' not found in available tools: {[tool['function']['name'] for tool in tools if tool['type'] == 'function']}" + ) + + # Parse parameters + param_matches = re.finditer(FN_PARAM_REGEX_PATTERN, fn_body, re.DOTALL) + params = _extract_and_validate_params( + matching_tool, param_matches, fn_name + ) + + # Create tool call with unique ID + tool_call_id = f'toolu_{tool_call_counter:02d}' + tool_call = { + 'index': 1, # always 1 because we only support **one tool call per message** + 'id': tool_call_id, + 'type': 'function', + 'function': {'name': fn_name, 'arguments': json.dumps(params)}, + } + tool_call_counter += 1 # Increment counter + + # Remove the function call part from content + if isinstance(content, list): + assert content and content[-1]['type'] == 'text' + content[-1]['text'] = ( + content[-1]['text'].split(' list[dict]: + """Break one message with multiple tool calls into multiple messages.""" + converted_messages = [] + + pending_tool_calls: dict[str, dict] = {} + for message in messages: + role, content = message['role'], message['content'] + if role == 'assistant': + if message.get('tool_calls') and len(message['tool_calls']) > 1: + # handle multiple tool calls by breaking them into multiple messages + for i, tool_call in enumerate(message['tool_calls']): + pending_tool_calls[tool_call['id']] = { + 'role': 'assistant', + 'content': content if i == 0 else '', + 'tool_calls': [tool_call], + } + else: + converted_messages.append(message) + elif role == 'tool': + if message['tool_call_id'] in pending_tool_calls: + # remove the tool call from the pending list + _tool_call_message = pending_tool_calls.pop(message['tool_call_id']) + converted_messages.append(_tool_call_message) + # add the tool result + converted_messages.append(message) + else: + assert ( + len(pending_tool_calls) == 0 + ), f'Found pending tool calls but not found in pending list: {pending_tool_calls=}' + converted_messages.append(message) + else: + assert ( + len(pending_tool_calls) == 0 + ), f'Found pending tool calls but not expect to handle it with role {role}: {pending_tool_calls=}, {message=}' + converted_messages.append(message) + + if not ignore_final_tool_result and len(pending_tool_calls) > 0: + raise FunctionCallConversionError( + f'Found pending tool calls but no tool result: {pending_tool_calls=}' + ) + return converted_messages + + + +def convert_fn_messages_to_non_fn_messages(messages: list[dict]) -> list[dict]: + """Convert function calling messages back to non-function calling messages.""" + new_messages = [] + for idx, message in enumerate(messages): + if message["role"] == "tool": + assert messages[idx-1]["role"] == "assistant" and new_messages[-1]["role"] == "assistant" + new_messages[-1]["content"] = messages[idx-1]["content"] + f""" +I have executed the tool {message["name"]} and the result is {message["content"]}. +""" + elif message["role"] == "assistant": + if message["tool_calls"] is not None: + msg_content = message["content"] + f""" +I want to use the tool named {message["tool_calls"][0]["function"]["name"]}, with the following arguments: {message["tool_calls"][0]["function"]["arguments"]}. +""" + else: + msg_content = message["content"] + new_messages.append({"role": message["role"], "content": msg_content}.copy()) + else: + new_messages.append(message.copy()) + return new_messages + + +def interleave_user_into_messages(messages: list[dict]) -> list[dict]: + new_messages = [] + for idx, message in enumerate(messages): + if message["role"] == "assistant" and messages[idx-1]["role"] == "assistant": + # 在两个连续的 assistant 消息之间插入一个空的 user 消息 + new_messages.append({ + "role": "user", + "content": "Please think twice and take the next action according to your previous actions and observations." # 空内容的用户消息 + }.copy()) + new_messages.append(message.copy()) + new_messages.append({"role": "user", "content": "Please think twice and take the next action according to your previous actions and observations."}) + return new_messages \ No newline at end of file diff --git a/autoagent/io_utils.py b/autoagent/io_utils.py new file mode 100644 index 0000000..4fe4faa --- /dev/null +++ b/autoagent/io_utils.py @@ -0,0 +1,51 @@ +import yaml +import hashlib +import zipfile +import os +import json +from rich.console import Console +def read_file(file_path): + with open(file_path, 'r', encoding='utf-8') as file: + content = file.read() + return content + +def read_yaml_file(file_path): + with open(file_path, 'r', encoding='utf-8') as file: + content = yaml.safe_load(file) + return content + +def get_file_md5(file_path): + md5_hash = hashlib.md5() + with open(file_path, "rb") as f: + # read file block + for byte_block in iter(lambda: f.read(4096), b""): + md5_hash.update(byte_block) + return md5_hash.hexdigest() + + +def compress_folder(source_folder, destination_folder, archive_name): + os.makedirs(destination_folder, exist_ok=True) + + archive_path = os.path.join(destination_folder, archive_name) + + with zipfile.ZipFile(archive_path, 'w', zipfile.ZIP_DEFLATED) as zipf: + + for root, _, files in os.walk(source_folder): + for file in files: + file_path = os.path.join(root, file) + arcname = os.path.relpath(file_path, source_folder) + zipf.write(file_path, arcname) + + print(f"Folder '{source_folder}' has been compressed to '{archive_path}'") + +def get_md5_hash_bytext(text): + return hashlib.md5(text.encode('utf-8')).hexdigest() + +def read_json_file(file_path): + with open(file_path, 'r', encoding='utf-8') as file: + content = json.load(file) + return content + +def print_stream(text): + console = Console() + console.print(f"[grey42]{text}[/grey42]") \ No newline at end of file diff --git a/autoagent/logger.py b/autoagent/logger.py new file mode 100644 index 0000000..193624c --- /dev/null +++ b/autoagent/logger.py @@ -0,0 +1,181 @@ +from datetime import datetime +from rich.console import Console +from rich.markup import escape +import json +from typing import List +from constant import DEBUG, DEFAULT_LOG, LOG_PATH, MC_MODE +from pathlib import Path +BAR_LENGTH = 60 +class MetaChainLogger: + def __init__(self, log_path: str): + self.log_path = log_path + self.console = Console() + self.debug = DEBUG + + def _write_log(self, message: str): + with open(self.log_path, 'a') as f: + f.write(message + '\n') + def _warp_args(self, args_dict: str): + args_dict = json.loads(args_dict) + args_str = '' + for k, v in args_dict.items(): + args_str += f"{repr(k)}={repr(v)}, " + return args_str[:-2] + def _wrap_title(self, title: str, color: str = None): + single_len = (BAR_LENGTH - len(title)) // 2 + color_bos = f"[{color}]" if color else "" + color_eos = f"[/{color}]" if color else "" + return f"{color_bos}{'*'*single_len} {title} {'*'*single_len}{color_eos}" + def info(self, *args: str, **kwargs: dict): + # console = Console() + timestamp = datetime.now().strftime("%Y-%m-%d %H:%M:%S") + message = "\n".join(map(str, args)) + color = kwargs.get("color", "white") + if MC_MODE: color = "grey58" + title = kwargs.get("title", "INFO") + log_str = f"[{timestamp}]\n{message}" + if self.debug: + # print_in_box(log_str, color=color, title=title) + self.console.print(self._wrap_title(title, f"bold {color}")) + print_str = escape(log_str) + if MC_MODE: print_str = f"[grey58]{print_str}[/grey58]" + self.console.print(print_str, highlight=True, emoji=True) + log_str = self._wrap_title(title) + "\n" + log_str + if self.log_path: self._write_log(log_str) + def lprint(self, *args: str, **kwargs: dict): + if not self.debug: return + timestamp = datetime.now().strftime("%Y-%m-%d %H:%M:%S") + message = "\n".join(map(str, args)) + color = kwargs.get("color", "white") + if MC_MODE: color = "grey58" + title = kwargs.get("title", "") + log_str = f"[{timestamp}]\n{message}" + # print_in_box(log_str, color=color, title=title) + self.console.print(self._wrap_title(title, f"bold {color}")) + print_str = escape(log_str) + if MC_MODE: print_str = f"[grey58]{print_str}[/grey58]" + self.console.print(print_str, highlight=True, emoji=True) + + def _wrap_timestamp(self, timestamp: str, color: bool = True): + color_bos = "[grey58]" if color else "" + color_eos = "[/grey58]" if color else "" + return f"{color_bos}[{timestamp}]{color_eos}" + def _print_tool_execution(self, message, timestamp: str): + if MC_MODE: colors = ["grey58"] * 3 + else: colors = ["pink3", "blue", "purple"] + self.console.print(self._wrap_title("Tool Execution", f"bold {colors[0]}")) + self.console.print(self._wrap_timestamp(timestamp, color=True)) + self.console.print(f"[bold {colors[1]}]Tool Execution:[/bold {colors[1]}]", end=" ") + self.console.print(f"[bold {colors[2]}]{message['name']}[/bold {colors[2]}]\n[bold {colors[1]}]Result:[/bold {colors[1]}]") + print_str = f"---\n{escape(message['content'])}\n---" + if MC_MODE: print_str = f"[grey58]{print_str}[/grey58]" + self.console.print(print_str, highlight=True, emoji=True) + def _save_tool_execution(self, message, timestamp: str): + self._write_log(self._wrap_title("Tool Execution")) + self._write_log(f"{self._wrap_timestamp(timestamp, color=False)}\ntool execution: {message['name']}\nResult:\n---\n{message['content']}\n---") + def _print_assistant_message(self, message, timestamp: str): + if MC_MODE: colors = ["grey58"] * 3 + else: colors = ["light_salmon3", "blue", "purple"] + self.console.print(self._wrap_title("Assistant Message", f"bold {colors[0]}")) + self.console.print(f"{self._wrap_timestamp(timestamp, color=True)}\n[bold {colors[1]}]{message['sender']}[/bold {colors[1]}]:", end=" ") + if message["content"]: + print_str = escape(message["content"]) + if MC_MODE: print_str = f"[grey58]{print_str}[/grey58]" + self.console.print(print_str, highlight=True, emoji=True) + else: + print_str = None + if MC_MODE: print_str = "[grey58]None[/grey58]" + self.console.print(print_str, highlight=True, emoji=True) + def _save_assistant_message(self, message, timestamp: str): + self._write_log(self._wrap_title("Assistant Message")) + content = message["content"] if message["content"] else None + self._write_log(f"{self._wrap_timestamp(timestamp, color=False)}\n{message['sender']}: {content}") + def _print_tool_call(self, tool_calls: List, timestamp: str): + if MC_MODE: colors = ["grey58"] * 3 + else: colors = ["light_pink1", "blue", "purple"] + if len(tool_calls) >= 1: self.console.print(self._wrap_title("Tool Calls", f"bold {colors[0]}")) + + for tool_call in tool_calls: + f = tool_call["function"] + name, args = f["name"], f["arguments"] + arg_str = self._warp_args(args) + print_arg_str = escape(arg_str) + if MC_MODE: print_arg_str = f"[grey58]{print_arg_str}[/grey58]" + self.console.print(f"{self._wrap_timestamp(timestamp, color=True)}\n[bold {colors[2]}]{name}[/bold {colors[2]}]({print_arg_str})") + def _save_tool_call(self, tool_calls: List, timestamp: str): + if len(tool_calls) >= 1: self._write_log(self._wrap_title("Tool Calls")) + + for tool_call in tool_calls: + f = tool_call["function"] + name, args = f["name"], f["arguments"] + arg_str = self._warp_args(args) + self._write_log(f"{self._wrap_timestamp(timestamp, color=False)}\n{name}({arg_str})") + + def pretty_print_messages(self, message, **kwargs) -> None: + # for message in messages: + if message["role"] != "assistant" and message["role"] != "tool": + return + # console = Console() + + # handle tool call + if message["role"] == "tool": + timestamp = datetime.now().strftime("%Y-%m-%d %H:%M:%S") + if self.log_path: self._save_tool_execution(message, timestamp) + if self.debug: self._print_tool_execution(message, timestamp) + return + + # handle assistant message + # print agent name in blue + timestamp = datetime.now().strftime("%Y-%m-%d %H:%M:%S") + if self.log_path: self._save_assistant_message(message, timestamp) + if self.debug: self._print_assistant_message(message, timestamp) + + # print tool calls in purple, if any + tool_calls = message.get("tool_calls") or [] + timestamp = datetime.now().strftime("%Y-%m-%d %H:%M:%S") + if self.log_path: self._save_tool_call(tool_calls, timestamp) + if self.debug: self._print_tool_call(tool_calls, timestamp) +class LoggerManager: + _instance = None + _logger: MetaChainLogger = None + + @classmethod + def get_instance(cls): + if cls._instance is None: + cls._instance = LoggerManager() + return cls._instance + + @classmethod + def get_logger(cls): + return cls.get_instance()._logger + + @classmethod + def set_logger(cls, new_logger): + cls.get_instance()._logger = new_logger +if DEFAULT_LOG: + if LOG_PATH is None: + log_dir = Path(f'logs/res_{datetime.now().strftime("%Y%m%d_%H%M%S")}') + log_dir.mkdir(parents=True, exist_ok=True) # recursively create all necessary parent directories + log_path = str(log_dir / "agent.log") + # logger = MetaChainLogger(log_path=log_path) + LoggerManager.set_logger(MetaChainLogger(log_path=log_path)) + else: + # logger = MetaChainLogger(log_path=LOG_PATH) + LoggerManager.set_logger(MetaChainLogger(log_path=LOG_PATH)) + # logger.info("Log file is saved to", logger.log_path, "...", title="Log Path", color="light_cyan3") + LoggerManager.get_logger().info("Log file is saved to", + LoggerManager.get_logger().log_path, "...", + title="Log Path", color="light_cyan3") +else: + # logger = None + LoggerManager.set_logger(None) +logger = LoggerManager.get_logger() + +def set_logger(new_logger): + LoggerManager.set_logger(new_logger) +# if __name__ == "__main__": +# logger = MetaChainLogger(log_path="test.log") +# logger.pretty_print_messages({"role": "assistant", "content": "Hello, world!", "tool_calls": [{"function": {"name": "test", "arguments": {"url": "https://www.google.com", "query": "test"}}}], "sender": "test_agent"}) + +# logger.pretty_print_messages({"role": "tool", "name": "test", "content": "import requests\n\nurl = 'https://www.google.com'\nquery = 'test'\n\nresponse = requests.get(url)\nprint(response.text)", "sender": "test_agent"}) +# logger.info("test content", color="red", title="test") diff --git a/autoagent/main.py b/autoagent/main.py new file mode 100644 index 0000000..2bd9e5c --- /dev/null +++ b/autoagent/main.py @@ -0,0 +1,109 @@ +from autoagent import MetaChain, Agent, Response +from typing import List +from autoagent.logger import MetaChainLogger +from autoagent.environment.utils import setup_metachain +from autoagent.environment.docker_env import DockerConfig, DockerEnv + +def case_resolved(result: str): + """ + Use this tool to indicate that the case is resolved. You can use this tool only after you truly resolve the case with exsiting tools and created new tools.Please encapsulate your final answer (answer ONLY) within and . + + Args: + result: The final result of the case resolution following the instructions. + + Example: case_resolved(`The answer to the question is 42 `) + """ + return f"Case resolved. No further actions are needed. The result of the case resolution is: {result}" + +def case_not_resolved(failure_reason: str, take_away_message: str): + """ + Use this tool to indicate that the case is not resolved when all agents have tried their best. + [IMPORTANT] Please do not use this function unless all of you have tried your best. + You should give the failure reason to tell the user why the case is not resolved, and give the take away message to tell which information you gain from creating new tools. + + Args: + failure_reason: The reason why the case is not resolved. + take_away_message: The message to take away from the case. + """ + return f"Case not resolved. The reason is: {failure_reason}. But though creating new tools, I gain some information: {take_away_message}" + +async def run_in_client( + agent: Agent, + messages: List, + context_variables: dict = {}, + logger: MetaChainLogger = None, + meta_agent: Agent = None, + docker_config: DockerConfig = None, + code_env: DockerConfig = None, +): + """ + """ + client = MetaChain(log_path=logger) + + MAX_RETRY = 3 + for i in range(MAX_RETRY): + try: + response: Response = await client.run_async(agent, messages, context_variables, debug=True) + except Exception as e: + logger.info(f'Exception in main loop: {e}', title='ERROR', color='red') + raise e + if 'Case resolved' in response.messages[-1]['content']: + break + elif 'Case not resolved' in response.messages[-1]['content']: + messages.extend(response.messages) + if meta_agent and (i >= 2): + setup_metachain(docker_config.workplace_name, code_env) + messages.append({ + 'role': 'user', + 'content': """\ +It seems that the case is not resolved with the existing agent system. +Help me to solve this problem by running tools in the MetaChain. +IMPORTANT: You should fully take advantage of existing tools, and if existing tools are not enough, you should develop new tools. +Use `visual_question_answering` tool for ALL visual tasks (images, videos, visual analysis, including object detection, etc.) +IMPORTANT: You can not stop with `case_not_resolved` after you try your best to creating new tools. +IMPORTANT: You should ONLY interact with the environment provided to you AND NEVER ASK FOR HUMAN HELP. +Please encapsulate your final answer (answer ONLY) within and . +""" + }) + meta_agent.functions.append(case_not_resolved) + meta_agent.functions.append(case_resolved) + response: Response = await client.run_async(meta_agent, messages, context_variables, debug=True) + if 'Case resolved' in response.messages[-1]['content']: + break + else: + messages.extend(response.messages) + + messages.append({ + 'role': 'user', + 'content': 'Please try to resolve the case again. It\'s important for me to resolve the case. Trying again in another way may be helpful.' + }) + + return response + +def run_in_client_non_async( + agent: Agent, + messages: List, + context_variables: dict = {}, + logger: MetaChainLogger = None, +): + """ + """ + client = MetaChain(log_path=logger) + + MAX_RETRY = 3 + for i in range(MAX_RETRY): + try: + response: Response = client.run(agent, messages, context_variables, debug=True) + except Exception as e: + logger.info(f'Exception in main loop: {e}', title='ERROR', color='red') + raise e + if 'Case resolved' in response.messages[-1]['content']: + break + elif 'Case not resolved' in response.messages[-1]['content']: + messages.extend(response.messages) + messages.append({ + 'role': 'user', + 'content': 'Please try to resolve the case again. It\'s important for me to resolve the case. Trying again in another way may be helpful.' + }) + + return response \ No newline at end of file diff --git a/autoagent/memory/__init__.py b/autoagent/memory/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/autoagent/memory/code_memory.py b/autoagent/memory/code_memory.py new file mode 100644 index 0000000..d89843c --- /dev/null +++ b/autoagent/memory/code_memory.py @@ -0,0 +1,174 @@ +import os +from typing import List, Dict +from autoagent.memory.rag_memory import Memory, Reranker +from litellm import completion +import re + +class CodeMemory(Memory): + def __init__(self, project_path: str, db_name: str = '.sa', platform: str = 'OpenAI', api_key: str = None, embedding_model: str = "text-embedding-ada-002"): + super().__init__(project_path, db_name, platform, api_key, embedding_model) + self.collection_name = 'code_memory' + + def add_code_files(self, directory: str, exclude_prefix: List[str] = ["workplace_"]): + """ + Add all code files in the specified directory to the memory. + + Args: + directory (str): The directory path containing the code files to add. + """ + code_files = [] + for root, _, files in os.walk(directory): + root_name = str(root) + if any(prefix in root_name for prefix in exclude_prefix): + continue + for file in files: + + if file.endswith(('.py', '.js', '.java', '.cpp', '.h', '.c', '.html', '.css')): # add more file types if needed + file_path = os.path.join(root, file) + with open(file_path, 'r', encoding='utf-8') as f: + content = f.read() + code_files.append({ + "query": f"File: {file_path}\n\nContent:\n{content}", + "response": f"This is the content of file {file_path}" + }) + self.add_query(code_files, self.collection_name) + + def query_code(self, query_text: str, n_results: int = 5) -> List[Dict]: + """ + Query the code memory. + + Args: + query_text (str): The query text + n_results (int): The number of results to return + + Returns: + List[Dict]: The query results list + """ + results = self.query([query_text], self.collection_name, n_results) + return [ + { + "file": doc.split('\n')[0].replace("File: ", ""), + "content": '\n'.join(doc.split('\n')[3:]), + "metadata": metadata + } + for doc, metadata in zip(results['documents'][0], results['metadatas'][0]) + ] + +class CodeReranker(Reranker): + def __init__(self, model: str) -> None: + super().__init__(model) + def wrap_query_results(self, query_results: List[Dict]) -> str: + wrapped_query_results = "" + for result in query_results: + wrapped_query_results += f"File: {result['file']}\n" + wrapped_query_results += f"Content: {result['content'][:300]}...\n" + wrapped_query_results += "---" + return wrapped_query_results + def wrap_reranked_results(self, reranked_paths: List[str]) -> str: + wrapped_reranked_results = "[Referenced code files]:" + for path in reranked_paths: + wrapped_reranked_results += f"Code path: {path}\n" + try: + with open(path, 'r', encoding='utf-8') as file: + content = file.read() + wrapped_reranked_results += f"Code content:\n{content}\n" + except Exception as e: + wrapped_reranked_results += f"Error reading file: {str(e)}\n" + wrapped_reranked_results += "---\n" + return wrapped_reranked_results + def parse_results(self, reranked_results: str) -> List[str]: + lines = reranked_results.strip().split('\n') + + # get the last 5 lines + last_lines = lines[-5:] + + # remove the number and dot at the beginning of each line + cleaned_lines = [re.sub(r'^\d+\.\s*', '', line.strip()) for line in last_lines] + unique_lines = list(dict.fromkeys(cleaned_lines)) + + return unique_lines + def rerank(self, query_text: str, query_results: List[Dict]) -> List[Dict]: + system_prompt = \ + """ + You are a helpful assistant that reranks the given code files (containing the path of files and Overview of the content of files) based on the query. + You should rerank the code files based on the query, and the most relevant code files should be ranked on the top. + You should select the top 5 code files to answer the query, by giving the file path of the code files. + + Example: + [Query]: "The definition of 'BaseAgent'" + [Code files]: + File: /Users/tangjiabin/Documents/reasoning/SelfAgent/sa/agents/__init__.py + Content: from .ABCAgent import ABCAgent + from .BaseAgent import BaseAgent + from .ManagerAgent import ManagerAge... + --- + File: /Users/tangjiabin/Documents/reasoning/SelfAgent/sa/agents/__init__.py + Content: from .ABCAgent import ABCAgent + from .BaseAgent import BaseAgent + from .ManagerAgent import ManagerAge... + --- + File: /Users/tangjiabin/Documents/reasoning/SelfAgent/sa/agents/__init__.py + Content: from .ABCAgent import ABCAgent + from .BaseAgent import BaseAgent + from .ManagerAgent import ManagerAge... + --- + File: /Users/tangjiabin/Documents/reasoning/SelfAgent/sa/agents/__init__.py + Content: from .ABCAgent import ABCAgent + from .BaseAgent import BaseAgent + from .ManagerAgent import ManagerAge... + --- + File: /Users/tangjiabin/Documents/reasoning/SelfAgent/sa/agent_prompts/__init__.py + Content: from .BasePrompt import BasePromptGen, ManagerPromptGen, PromptGen + ... + --- + File: /Users/tangjiabin/Documents/reasoning/SelfAgent/sa/agent_prompts/__init__.py + Content: from .BasePrompt import BasePromptGen, ManagerPromptGen, PromptGen + ... + --- + File: /Users/tangjiabin/Documents/reasoning/SelfAgent/sa/agent_prompts/__init__.py + Content: from .BasePrompt import BasePromptGen, ManagerPromptGen, PromptGen + ... + --- + File: /Users/tangjiabin/Documents/reasoning/SelfAgent/sa/agent_prompts/__init__.py + Content: from .BasePrompt import BasePromptGen, ManagerPromptGen, PromptGen + ... + --- + File: /Users/tangjiabin/Documents/reasoning/SelfAgent/sa/agents/BaseAgent.py + Content: from typing import List + + from sa.actions import BaseAction, FinishAct, ThinkAct, PlanAct + from sa.age... + --- + File: /Users/tangjiabin/Documents/reasoning/SelfAgent/sa/agents/BaseAgent.py + Content: from typing import List + + from sa.actions import BaseAction, FinishAct, ThinkAct, PlanAct + from sa.age... + --- + [Reranked 5 code files]: + 1. /Users/tangjiabin/Documents/reasoning/SelfAgent/sa/agents/BaseAgent.py + 2. /Users/tangjiabin/Documents/reasoning/SelfAgent/sa/agents/__init__.py + 3. /Users/tangjiabin/Documents/reasoning/SelfAgent/sa/agents/ABCAgent.py + 4. /Users/tangjiabin/Documents/reasoning/SelfAgent/sa/agents/ManagerAgent.py + 5. /Users/tangjiabin/Documents/reasoning/SelfAgent/sa/agents/AgentLogger.py + """ + wrapped_query_results = self.wrap_query_results(query_results) + user_prompt = \ + """ + [Query]: \n{query_text} + [Code files]: \n{query_results} + [Reranked 5 code files]: + """.format(query_text=query_text, query_results=wrapped_query_results) + chat_history = [ + {"role": "system", "content": system_prompt}, + {"role": "user", "content": user_prompt} + ] + create_params = { + "model": self.model, + "messages": chat_history, + "stream": False, + } + response = completion(**create_params) + reranked_results = self.parse_results(response.choices[0].message.content) + reranked_results = self.wrap_reranked_results(reranked_results) + return reranked_results diff --git a/autoagent/memory/code_tree/code_parser.py b/autoagent/memory/code_tree/code_parser.py new file mode 100644 index 0000000..3334763 --- /dev/null +++ b/autoagent/memory/code_tree/code_parser.py @@ -0,0 +1,123 @@ +import dataclasses +from tree_sitter import Language +import tree_sitter +import glob +import uuid +from loguru import logger + +@dataclasses.dataclass +class Snippet: + """Dataclass for storing Embedded Snippets""" + + id: str + embedding: list[float] | None + snippet: str + filename: str + language: str + + +class CodeParser: + """Code Parser Class.""" + + def __init__(self, language: str, node_types: list[str], path_to_object_file: str): + self.node_types = node_types + self.language = language + try: + self.parser = tree_sitter.Parser() + self.parser.set_language( + tree_sitter.Language(f"{path_to_object_file}/my-languages.so", language) + ) + except Exception as e: + logger.exception("failed to build %s parser: ", e) + + def parse_file(self, content: str, filename: str): + """ + Parse code snippets from single code file. + + Args: + content: The content of the file. + filename: The name of the code file. + + Returns: + List of Parsed Snippets + """ + try: + tree = self.parser.parse(content) + except Exception as e: + logger.error(f"Failed to parse snippet: {filename} \n Error: {e}") + return + + cursor = tree.walk() + parsed_snippets = [] + + # Walking nodes from abstract syntax tree + while cursor.goto_first_child(): + if cursor.node.type in self.node_types: + parsed_snippets.append( + Snippet( + id=str(uuid.uuid4()), + snippet=cursor.node.text, + filename=filename, + language=self.language, + embedding=None, + ) + ) + + while cursor.goto_next_sibling(): + if cursor.node.type in self.node_types: + parsed_snippets.append( + Snippet( + id=str(uuid.uuid4()), + snippet=cursor.node.text, + filename=filename, + language=self.language, + embedding=None, + ) + ) + return parsed_snippets + + def parse_directory(self, code_directory_path): + """ + Parse code snippets from all files in directory. + + Args: + code_directory_path: Directory path containing code files. + + Returns: + List of Parsed Snippets + """ + parsed_contents = [] + for filename in glob.glob(f"{code_directory_path}/**/*.py", recursive=True): + # print(filename) + with open(filename, "rb") as codefile: + code_content = codefile.read() + + parsed_content = self.parse_file(code_content, filename) + parsed_contents.extend(parsed_content) + + return parsed_contents + +def to_dataframe_row(embedded_snippets: list[Snippet]): + """ + Helper function to convert Embedded Snippet object to a dataframe row + in dictionary format. + + Args: + embedded_snippets: List of Snippets to be converted + + Returns: + List of Dictionaries + """ + outputs = [] + for embedded_snippet in embedded_snippets: + output = { + "ids": embedded_snippet.id, + "embeddings": embedded_snippet.embedding, + "snippets": embedded_snippet.snippet, + "metadatas": { + "filenames": embedded_snippet.filename, + "languages": embedded_snippet.language, + }, + } + outputs.append(output) + return outputs \ No newline at end of file diff --git a/autoagent/memory/codetree_memory.py b/autoagent/memory/codetree_memory.py new file mode 100644 index 0000000..076acaa --- /dev/null +++ b/autoagent/memory/codetree_memory.py @@ -0,0 +1,118 @@ +import os +from typing import List, Dict +from autoagent.memory.rag_memory import Memory, Reranker +import openai +import re +from autoagent.memory.code_tree.code_parser import CodeParser, to_dataframe_row +from tree_sitter import Language +from loguru import logger +from openai import OpenAI +import pandas as pd +class CodeTreeMemory(Memory): + def __init__(self, project_path: str, db_name: str = '.code_tree', platform: str = 'OpenAI', api_key: str = None, embedding_model: str = "text-embedding-ada-002"): + super().__init__(project_path, db_name, platform, api_key, embedding_model) + self.collection_name = 'code_tree_memory' + self.embedder = OpenAI(api_key=api_key) + + + def add_code_files(self, directory: str, exclude_prefix: List[str] = ["workplace_"]): + """ + 将指定目录下的所有代码文件添加到内存中。 + + Args: + directory (str): 要添加的代码文件所在的目录路径 + """ + tree_sitter_parent_dir = os.path.dirname(os.getcwd()) + # Build Tree sitter Parser object + Language.build_library( + f"{tree_sitter_parent_dir}/my-languages.so", + [ + f"{tree_sitter_parent_dir}/tree-sitter-python", + ], + ) + parser = CodeParser( + language="python", + node_types=["class_definition", "function_definition"], + path_to_object_file=tree_sitter_parent_dir, + ) + logger.info("Parsing Code...") + parsed_snippets = parser.parse_directory( + directory + ) + snippet_texts = list(map(lambda x: x.snippet.decode("ISO-8859-1"), parsed_snippets)) + embedded_texts = self.embedder.embeddings.create(input=snippet_texts, model="text-embedding-3-small").data + embedded_snippets = [] + for code_text, embedding, snippet in zip( + snippet_texts, embedded_texts, parsed_snippets + ): + snippet.snippet = code_text + snippet.embedding = embedding.embedding + embedded_snippets.append(snippet) + + # Convert Snippets to DataFrame for ChromaDB Ingestion + data = pd.DataFrame(to_dataframe_row(embedded_snippets)) + collection = self.client.get_or_create_collection( + name=self.collection_name, metadata={"hnsw:space": "cosine"} + ) + logger.info( + f"Adding {data.shape[0]} Code snippets and embedding to " + "local chroma db collection..." + ) + collection.add( + documents=data["snippets"].tolist(), + embeddings=data["embeddings"].tolist(), + metadatas=data["metadatas"].tolist(), + ids=data["ids"].tolist(), + ) + + + def query_code(self, query_text: str, n_results: int = 5) -> List[Dict]: + """ + Query the code memory. + + Args: + query_text (str): The query text + n_results (int): The number of results to return + + Returns: + List[Dict]: The query results list + """ + query_embedding = self.embedder.embeddings.create(input=[query_text], model="text-embedding-3-small").data[0].embedding + results = self.client.get_or_create_collection(self.collection_name).query(query_embeddings=[query_embedding], n_results=n_results) + return [ + { + "file": metadata['filenames'], + "content": doc + } + for doc, metadata in zip(results['documents'][0], results['metadatas'][0]) + ] +class DummyReranker(Reranker): + def __init__(self, model: str = None) -> None: + super().__init__(model) + def rerank(self, query_results: List[Dict]) -> List[Dict]: + wrapped_reranked_results = "[Referenced code files]:" + result_path = [] + for result in query_results: + if result['file'] in result_path: + continue + else: + result_path.append(result['file']) + wrapped_reranked_results = f"Code path: {result['file']}\n" + wrapped_reranked_results += f"Code content:\n{result['content']}...\n" + wrapped_reranked_results += "---\n" + return wrapped_reranked_results + +# 使用示例 +if __name__ == "__main__": + code_memory = CodeTreeMemory(project_path = './code_db', db_name='code_tree', platform='OpenAI', api_key='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') + + # 添加代码文件到内存 + code_memory.add_code_files("/Users/tangjiabin/Documents/reasoning/SelfAgent/workplace_test/SelfAgent", exclude_prefix=['workplace_', '__pycache__', 'code_db', '.git']) + + # 查询代码 + query_results = code_memory.query_code("The definition of BaseAction", n_results=10) + + for result in query_results: + print(f"File: {result['file']}") + print(f"Content: {result['content'][:100]}...") # 只打印前100个字符 + print("---") \ No newline at end of file diff --git a/autoagent/memory/paper_memory.py b/autoagent/memory/paper_memory.py new file mode 100644 index 0000000..b4d2479 --- /dev/null +++ b/autoagent/memory/paper_memory.py @@ -0,0 +1,80 @@ +import pandas as pd +from typing import List, Dict +from autoagent.memory.rag_memory import Memory, Reranker +import json +import math +import os +from litellm import completion +from autoagent.memory.utils import chunking_by_token_size + +class TextMemory(Memory): + def __init__( + self, + project_path: str, + db_name: str = '.text_table', + platform: str = 'OpenAI', + api_key: str = None, + embedding_model: str = "text-embedding-3-small", + ): + super().__init__( + project_path=project_path, + db_name=db_name, + platform=platform, + api_key=api_key, + embedding_model=embedding_model + ) + self.collection_name = 'text_memory' + + def add_text_content(self, paper_content: str, batch_size: int = 100, collection = None): + assert collection is not None, "Collection is required. Should be the path of the paper." + queries = [] + content_chunks = chunking_by_token_size(paper_content, max_token_size=4096) + + idx_list = ["chunk_" + str(chunk['chunk_order_index']) for chunk in content_chunks] + for chunk in content_chunks: + query = { + 'query': chunk['content'], + 'response': chunk['content'] + } + queries.append(query) + + # self.add_query(queries, collection=collection) + print(f'Adding {len(queries)} queries to {collection} with batch size {batch_size}') + num_batches = math.ceil(len(queries) / batch_size) + + for i in range(num_batches): + start_idx = i * batch_size + end_idx = min((i + 1) * batch_size, len(queries)) + batch_queries = queries[start_idx:end_idx] + batch_idx = idx_list[start_idx:end_idx] + + # Add the current batch of queries + self.add_query(batch_queries, collection=collection, idx=batch_idx) + + print(f"Batch {i+1}/{num_batches} added") + + def query_text_content( + self, + query_text: str, + collection: str = None, + n_results: int = 5 + ) -> List[str]: + """ + Query the table and return the results + """ + assert collection is not None, "Collection is required. Should be the path of the paper." + results = self.query([query_text], collection=collection, n_results=n_results) + + metadata_results = results['metadatas'][0] + + results = [item['response'] for item in metadata_results] + return results + + def peek_table(self, collection: str = None, n_results: int = 20) -> pd.DataFrame: + """ + Peek at the data in the table + """ + assert collection is not None, "Collection is required. Should be the path of the paper." + raw_results = self.peek(collection=collection, n_results=n_results) + results = [item['response'] for item in raw_results['metadatas']] + return results \ No newline at end of file diff --git a/autoagent/memory/rag_memory.py b/autoagent/memory/rag_memory.py new file mode 100644 index 0000000..a18be83 --- /dev/null +++ b/autoagent/memory/rag_memory.py @@ -0,0 +1,179 @@ +import uuid +import os.path +from datetime import datetime +from typing import List, Dict +import chromadb +from chromadb.utils import embedding_functions +from abc import ABC, abstractmethod +from openai import OpenAI +import numpy as np +from chromadb.api.types import QueryResult +chromadb.logger.setLevel(chromadb.logging.ERROR) + + +class Memory: + def __init__( + self, + project_path: str, + db_name: str = '.sa', + platform: str = 'OpenAI', + api_key: str = None, + embedding_model: str = "text-embedding-3-small" + ): + """ + Memory: memory and external knowledge management. + Args: + project_path: the path to store the data. + embedding_model: the embedding model to use, default will use the embedding model from ChromaDB, + if the OpenAI has been set in the configuration, it will use the OpenAI embedding model + "text-embedding-ada-002". + """ + self.db_name = db_name + self.collection_name = 'memory' + self.client = chromadb.PersistentClient(path=os.path.join(project_path, self.db_name)) + self.client.get_or_create_collection( + self.collection_name, + ) + # use the OpenAI embedding function if the openai section is set in the configuration. + if platform == 'OpenAI': + openai_client = OpenAI(api_key=api_key or os.environ["OPENAI_API_KEY"]) + self.embedder = lambda x: [i.embedding for i in openai_client.embeddings.create(input=x, model=embedding_model).data] + else: + # self.embedder = embedding_functions.DefaultEmbeddingFunction() + self.embedder = embedding_functions.SentenceTransformerEmbeddingFunction(model_name="all-MiniLM-L6-v2") + + def add_query( + self, + queries: List[Dict[str, str]], + collection: str = None, + idx: List[str] = None + ): + """ + add_query: add the queries to the memery. + Args: + queries: the queries to add to the memery. Should be in the format of + { + "query": "the query", + "response": "the response" + } + collection: the name of the collection to add the queries. + idx: the ids of the queries, should be in the same length as the queries. + If not provided, the ids will be generated by UUID. + + Return: A list of generated IDs. + """ + if idx: + ids = idx + else: + ids = [str(uuid.uuid4()) for _ in range(len(queries))] + + if not collection: + collection = self.collection_name + + query_list = [query['query'] for query in queries] + embeddings = self.embedder(query_list) + added_time = datetime.now().isoformat() + resp_list = [{'response': query['response'], 'created_at': added_time} for query in queries] + # insert the record into the database + self.client.get_or_create_collection(collection).add( + documents=query_list, + metadatas=resp_list, + ids=ids, + embeddings=embeddings + ) + + return ids + + def query(self, query_texts: List[str], collection: str = None, n_results: int = 5) -> QueryResult: + """ + query: query the memery. + Args: + query_texts: the query texts to search in the memery. + collection: the name of the collection to search. + n_results: the number of results to return. + + Returns: QueryResult + class QueryResult(TypedDict): + ids: List[IDs] + embeddings: Optional[ + Union[ + List[Embeddings], + List[PyEmbeddings], + List[NDArray[Union[np.int32, np.float32]]], + ] + ] + documents: Optional[List[List[Document]]] + uris: Optional[List[List[URI]]] + data: Optional[List[Loadable]] + metadatas: Optional[List[List[Metadata]]] + distances: Optional[List[List[float]]] + included: Include + """ + if not collection: + collection = self.collection_name + query_embedding = self.embedder(query_texts) + return self.client.get_or_create_collection(collection).query(query_embeddings=query_embedding, n_results=n_results) + + def peek(self, collection: str = None, n_results: int = 20): + """ + peek: peek the memery. + Args: + collection: the name of the collection to peek. + n_results: the number of results to return. + + Returns: the top k results. + """ + if not collection: + collection = self.collection_name + return self.client.get_or_create_collection(collection).peek(limit=n_results) + + def get(self, collection: str = None, record_id: str = None): + """ + get: get the record by the id. + Args: + record_id: the id of the record. + collection: the name of the collection to get the record. + + Returns: the record. + """ + if not collection: + collection = self.collection_name + collection = self.client.get_collection(collection) + if not record_id: + return collection.get() + + return collection.get(record_id) + + def delete(self, collection_name=None): + """ + delete: delete the memery collections. + Args: + collection_name: the name of the collection to delete. + """ + if not collection_name: + collection_name = self.collection_name + return self.client.delete_collection(name=collection_name) + + def count(self, collection_name=None): + """ + count: count the number of records in the memery. + Args: + collection_name: the name of the collection to count. + """ + if not collection_name: + collection_name = self.collection_name + return self.client.get_or_create_collection(name=collection_name).count() + + def reset(self): + """ + reset: reset the memory. + Notice: You may need to set the environment variable `ALLOW_RESET` to `TRUE` to enable this function. + """ + self.client.reset() + +class Reranker: + def __init__(self, model: str) -> None: + self.model = model + @abstractmethod + def rerank(self, query_text: str, query_results: List[Dict]) -> List[Dict]: + raise NotImplementedError("Reranker is not implemented") \ No newline at end of file diff --git a/autoagent/memory/tool_memory.py b/autoagent/memory/tool_memory.py new file mode 100644 index 0000000..03eff95 --- /dev/null +++ b/autoagent/memory/tool_memory.py @@ -0,0 +1,164 @@ +import pandas as pd +from typing import List, Dict +from autoagent.memory.rag_memory import Memory, Reranker +import json +import math +import os +from litellm import completion +from pydantic import BaseModel +""" +Category | Tool_Name | Tool_Description | API_Name | API_Description | Method | API_Details | Required_API_Key | Platform +""" +class ToolMemory(Memory): + def __init__( + self, + project_path: str, + db_name: str = '.tool_table', + platform: str = 'OpenAI', + api_key: str = None, + embedding_model: str = "text-embedding-3-small", + ): + super().__init__( + project_path=project_path, + db_name=db_name, + platform=platform, + api_key=api_key, + embedding_model=embedding_model + ) + self.collection_name = 'tool_memory' + + def add_dataframe(self, df: pd.DataFrame, collection: str = None, batch_size: int = 100): + if not collection: + collection = self.collection_name + queries = [] + for idx, row in df.iterrows(): + query = { + 'query': ' '.join(row[['Tool_Name', 'Tool_Description', 'API_Name', 'API_Description']].astype(str)), + 'response': row.to_json() + } + queries.append(query) + + # self.add_query(queries, collection=collection) + print(f'Adding {len(queries)} queries to {collection} with batch size {batch_size}') + num_batches = math.ceil(len(queries) / batch_size) + + for i in range(num_batches): + start_idx = i * batch_size + end_idx = min((i + 1) * batch_size, len(queries)) + batch_queries = queries[start_idx:end_idx] + + # Add the current batch of queries + self.add_query(batch_queries, collection=collection) + + print(f"Batch {i+1}/{num_batches} added") + + def query_table( + self, + query_text: str, + collection: str = None, + n_results: int = 5 + ) -> pd.DataFrame: + """ + Query the table and return the results + """ + if not collection: + collection = self.collection_name + results = self.query([query_text], collection=collection, n_results=n_results) + + metadata_results = results['metadatas'][0] + + df_results = pd.DataFrame([json.loads(item['response']) for item in metadata_results]) + return df_results + + def peek_table(self, collection: str = None, n_results: int = 20) -> pd.DataFrame: + """ + Peek at the data in the table + """ + if not collection: + collection = self.collection_name + results = self.peek(collection=collection, n_results=n_results) + df_results = pd.DataFrame([json.loads(item['response']) for item in results['metadatas']]) + return df_results + +class ToolReranker(Reranker): + def rerank(self, query_text: str, query_df: pd.DataFrame) -> str: + system_prompt = \ + """ + You are a helpful assistant that reranks the given API table based on the query. + You should select the top 5 APIs to answer the query in the given format. + You can only select APIs I give you. + Directly give the answer without any other words. + """ + # Use the DataFrame's to_dict method to convert all rows to a list of dictionaries + # print('query_df', query_df) + api_data = query_df.to_dict(orient='records') + + # Use a list comprehension and f-string to format each API's data + api_prompts = [f"\n\nAPI {i+1}:\n{api}" for i, api in enumerate(api_data)] + + # add the query text to the prompt + prompt = ''.join(api_prompts) + prompt = f"The query is: {query_text}\n\n{prompt}" + message = [ + {"role": "system", "content": system_prompt}, + {"role": "user", "content": prompt} + ] + + class Tools(BaseModel): + tool_name: str + api_name: str + rank: int + + class RerankResult(BaseModel): + tools: list[Tools] + + create_params = { + "model": self.model, + "messages": message, + "stream": False, + "response_format": RerankResult + } + response = completion(**create_params).choices[0].message.content + print(response) + rerank_result = json.loads(response)["tools"] + print(rerank_result) + if len(rerank_result) == 0: + return "Fail to retrieve the relevant information from the tool documentation." + try: + return self.wrap_rerank_result(rerank_result, query_df) + except Exception as e: + raise ValueError(f"Failed to wrap rerank result: {e}") + + def wrap_rerank_result(self, rerank_result: List[pd.DataFrame], query_df: pd.DataFrame) -> str: + res = "" + res_tmp = """ +The rank {rank} referenced tool documentation is: +API Name: {api_name} +API Description: {api_description} +API Details: {api_details} +Required API Key: {required_api_key} +Platform: {platform} +""" + try: + for tool_api in rerank_result: + tool_name = tool_api['tool_name'] + api_name = tool_api['api_name'] + matched_rows = query_df[(query_df['API_Name'] == api_name) & (query_df['Tool_Name'] == tool_name)] + if not matched_rows.empty: + res = res + res_tmp.format(rank=tool_api['rank'], api_name=matched_rows['API_Name'].values[0], api_description=matched_rows['API_Description'].values[0], api_details=matched_rows['API_Details'].values[0], required_api_key=matched_rows['Required_API_Key'].values[0], platform=matched_rows['Platform'].values[0]) + return res + except Exception as e: + raise ValueError(f"Failed to wrap rerank result: {e}") + def dummy_rerank(self, query_text: str, query_df: pd.DataFrame) -> str: + res = "" + res_tmp = """ +The rank {rank} referenced tool documentation is: +API Name: {api_name} +API Description: {api_description} +API Details: {api_details} +Required API Key: {required_api_key} +Platform: {platform} +""" + for i in range(len(query_df)): + res = res + res_tmp.format(rank=i+1, api_name=query_df['API_Name'].values[i], api_description=query_df['API_Description'].values[i], api_details=query_df['API_Details'].values[i], required_api_key=query_df['Required_API_Key'].values[i], platform=query_df['Platform'].values[i]) + return res diff --git a/autoagent/memory/utils.py b/autoagent/memory/utils.py new file mode 100644 index 0000000..4d0657f --- /dev/null +++ b/autoagent/memory/utils.py @@ -0,0 +1,36 @@ +import tiktoken +ENCODER = None + +def encode_string_by_tiktoken(content: str, model_name: str = "gpt-4o"): + global ENCODER + if ENCODER is None: + ENCODER = tiktoken.encoding_for_model(model_name) + tokens = ENCODER.encode(content) + return tokens + + +def decode_tokens_by_tiktoken(tokens: list[int], model_name: str = "gpt-4o"): + global ENCODER + if ENCODER is None: + ENCODER = tiktoken.encoding_for_model(model_name) + content = ENCODER.decode(tokens) + return content +def chunking_by_token_size( + content: str, overlap_token_size=128, max_token_size=1024, tiktoken_model="gpt-4o" +): + tokens = encode_string_by_tiktoken(content, model_name=tiktoken_model) + results = [] + for index, start in enumerate( + range(0, len(tokens), max_token_size - overlap_token_size) + ): + chunk_content = decode_tokens_by_tiktoken( + tokens[start : start + max_token_size], model_name=tiktoken_model + ) + results.append( + { + "tokens": min(max_token_size, len(tokens) - start), + "content": chunk_content.strip(), + "chunk_order_index": index, + } + ) + return results \ No newline at end of file diff --git a/autoagent/registry.py b/autoagent/registry.py new file mode 100644 index 0000000..ad67bab --- /dev/null +++ b/autoagent/registry.py @@ -0,0 +1,226 @@ +from typing import Callable, Dict, Any, Union, Literal, List, Optional +from dataclasses import dataclass, asdict +import inspect +import os +import functools +import tiktoken +MAX_OUTPUT_LENGTH = 12000 + +def encode_string_by_tiktoken(content: str, model_name: str = "gpt-4o"): + ENCODER = tiktoken.encoding_for_model(model_name) + tokens = ENCODER.encode(content) + return tokens + + +def decode_tokens_by_tiktoken(tokens: list[int], model_name: str = "gpt-4o"): + ENCODER = tiktoken.encoding_for_model(model_name) + content = ENCODER.decode(tokens) + return content +def truncate_output(output: str, max_length: int = MAX_OUTPUT_LENGTH) -> str: + """Truncate output if it exceeds max_length""" + tokens = encode_string_by_tiktoken(output) + if len(tokens) > max_length: + return decode_tokens_by_tiktoken(tokens[:max_length]) + f"\n\n[TOOL WARNING] Output truncated, exceeded {max_length} tokens)\n[TOOL SUGGESTION] Maybe this tool with direct output is not an optimal choice, consider save the output to a file in the `workplace/` directory to implement the same functionality." + return output + +@dataclass +class FunctionInfo: + name: str + func_name: str + func: Callable + args: List[str] + docstring: Optional[str] + body: str + return_type: Optional[str] + file_path: Optional[str] + def to_dict(self) -> dict: + # using asdict, but exclude func field because it cannot be serialized + d = asdict(self) + d.pop('func') # remove func field + return d + + @classmethod + def from_dict(cls, data: dict) -> 'FunctionInfo': + # if you need to create an object from a dictionary + if 'func' not in data: + data['func'] = None # or other default value + return cls(**data) +class Registry: + _instance = None + _registry: Dict[str, Dict[str, Callable]] = { + "tools": {}, + "agents": {}, + "plugin_tools": {}, + "plugin_agents": {}, + "workflows": {} + } + _registry_info: Dict[str, Dict[str, FunctionInfo]] = { + "tools": {}, + "agents": {}, + "plugin_tools": {}, + "plugin_agents": {}, + "workflows": {} + } + + def __new__(cls): + if cls._instance is None: + cls._instance = super().__new__(cls) + return cls._instance + + def register(self, + type: Literal["tool", "agent", "plugin_tool", "plugin_agent", "workflow"], + name: str = None, + func_name: str = None): + """ + 统一的注册装饰器 + Args: + type: 注册类型，"tool" 或 "agent" + name: 可选的注册名称 + """ + def decorator(func: Callable): + nonlocal name + if name is None: + name = func.__name__ + # if type == "agent" and name.startswith('get_'): + # name = name[4:] # 对 agent 移除 'get_' 前缀 + # 获取函数的文件路径 + + if type == "plugin_tool": + original_func = func # 保存原始函数 + @functools.wraps(original_func) + def wrapped_func(*args, **kwargs): + result = original_func(*args, **kwargs) # 调用原始函数 + if isinstance(result, str): + return truncate_output(result) + return result + else: + wrapped_func = func + try: + file_path = os.path.abspath(inspect.getfile(func)) + except: + file_path = "Unknown" + + # 获取函数信息 + signature = inspect.signature(func) + args = list(signature.parameters.keys()) + docstring = inspect.getdoc(func) + + # 获取函数体 + source_lines = inspect.getsource(func) + # 移除装饰器和函数定义行 + body_lines = source_lines.split('\n')[1:] # 跳过装饰器行 + while body_lines and (body_lines[0].strip().startswith('@') or 'def ' in body_lines[0]): + body_lines = body_lines[1:] + body = '\n'.join(body_lines) + + # 获取返回类型提示 + return_type = None + if signature.return_annotation != inspect.Signature.empty: + return_type = str(signature.return_annotation) + + # 创建函数信息对象 + func_info = FunctionInfo( + name=name, + func_name=func_name, + func=wrapped_func if type == "plugin_tool" else func, + args=args, + docstring=docstring, + body=body, + return_type=return_type, + file_path=file_path # 添加文件路径 + ) + + registry_type = f"{type}s" + self._registry[registry_type][func_name] = wrapped_func if type == "plugin_tool" else func + self._registry_info[registry_type][name] = func_info + return wrapped_func if type == "plugin_tool" else func + return decorator + + @property + def tools(self) -> Dict[str, Callable]: + return self._registry["tools"] + + @property + def agents(self) -> Dict[str, Callable]: + return self._registry["agents"] + + @property + def plugin_tools(self) -> Dict[str, Callable]: + return self._registry["plugin_tools"] + + @property + def plugin_agents(self) -> Dict[str, Callable]: + return self._registry["plugin_agents"] + + @property + def workflows(self) -> Dict[str, Callable]: + return self._registry["workflows"] + + @property + def tools_info(self) -> Dict[str, FunctionInfo]: + return self._registry_info["tools"] + + @property + def agents_info(self) -> Dict[str, FunctionInfo]: + return self._registry_info["agents"] + + @property + def plugin_tools_info(self) -> Dict[str, FunctionInfo]: + return self._registry_info["plugin_tools"] + + @property + def plugin_agents_info(self) -> Dict[str, FunctionInfo]: + return self._registry_info["plugin_agents"] + + @property + def workflows_info(self) -> Dict[str, FunctionInfo]: + return self._registry_info["workflows"] + + @property + def display_plugin_tools_info(self): + display_info = {} + for name, info in self.plugin_tools_info.items(): + tmp_info = info.to_dict().copy() + tmp_info.pop('func', None) + display_info[name] = tmp_info + return display_info + + @property + def display_plugin_agents_info(self): + display_info = {} + for name, info in self.plugin_agents_info.items(): + tmp_info = info.to_dict().copy() + tmp_info.pop('func', None) + display_info[name] = tmp_info + return display_info + + @property + def display_workflows_info(self): + display_info = {} + for name, info in self.workflows_info.items(): + tmp_info = info.to_dict().copy() + tmp_info.pop('func', None) + display_info[name] = tmp_info + return display_info + +# 创建全局实例 +registry = Registry() + +# 便捷的注册函数 +def register_tool(name: str = None): + func_name = name + return registry.register(type="tool", name=name, func_name=func_name) + +def register_agent(name: str = None, func_name: str = None): + return registry.register(type="agent", name=name, func_name=func_name) + +def register_plugin_tool(name: str = None): + func_name = name + return registry.register(type="plugin_tool", name=name, func_name=func_name) + +def register_plugin_agent(name: str = None, func_name: str = None): + return registry.register(type="plugin_agent", name=name, func_name=func_name) + +def register_workflow(name: str = None): + func_name = name + return registry.register(type="workflow", name=name, func_name=func_name) \ No newline at end of file diff --git a/autoagent/repl/__init__.py b/autoagent/repl/__init__.py new file mode 100644 index 0000000..2a1cb40 --- /dev/null +++ b/autoagent/repl/__init__.py @@ -0,0 +1 @@ +from .repl import run_demo_loop diff --git a/autoagent/repl/repl.py b/autoagent/repl/repl.py new file mode 100644 index 0000000..9868a15 --- /dev/null +++ b/autoagent/repl/repl.py @@ -0,0 +1,87 @@ +import json + +from autoagent.core import MetaChain + + +def process_and_print_streaming_response(response): + content = "" + last_sender = "" + + for chunk in response: + if "sender" in chunk: + last_sender = chunk["sender"] + + if "content" in chunk and chunk["content"] is not None: + if not content and last_sender: + print(f"\033[94m{last_sender}:\033[0m", end=" ", flush=True) + last_sender = "" + print(chunk["content"], end="", flush=True) + content += chunk["content"] + + if "tool_calls" in chunk and chunk["tool_calls"] is not None: + for tool_call in chunk["tool_calls"]: + f = tool_call["function"] + name = f["name"] + if not name: + continue + print(f"\033[94m{last_sender}: \033[95m{name}\033[0m()") + + if "delim" in chunk and chunk["delim"] == "end" and content: + print() # End of response message + content = "" + + if "response" in chunk: + return chunk["response"] + + +def pretty_print_messages(messages) -> None: + for message in messages: + if message["role"] != "assistant": + continue + + # print agent name in blue + print(f"\033[94m{message['sender']}\033[0m:", end=" ") + + # print response, if any + if message["content"]: + print(message["content"]) + + # print tool calls in purple, if any + tool_calls = message.get("tool_calls") or [] + if len(tool_calls) > 1: + print() + for tool_call in tool_calls: + f = tool_call["function"] + name, args = f["name"], f["arguments"] + arg_str = json.dumps(json.loads(args)).replace(":", "=") + print(f"\033[95m{name}\033[0m({arg_str[1:-1]})") + + +def run_demo_loop( + starting_agent, context_variables=None, stream=False, debug=False +) -> None: + client = MetaChain() + print("Starting Swarm CLI 🐝") + + messages = [] + agent = starting_agent + + while True: + user_input = input("\033[90mUser\033[0m: ") + messages.append({"role": "user", "content": user_input}) + + response = client.run( + agent=agent, + messages=messages, + context_variables=context_variables or {}, + stream=stream, + debug=debug, + ) + + if stream: + response = process_and_print_streaming_response(response) + else: + pretty_print_messages(response.messages) + + messages.extend(response.messages) + agent = response.agent diff --git a/autoagent/server.py b/autoagent/server.py new file mode 100644 index 0000000..c82e108 --- /dev/null +++ b/autoagent/server.py @@ -0,0 +1,144 @@ +from fastapi import FastAPI, HTTPException +from pydantic import BaseModel +from contextlib import asynccontextmanager +from typing import Dict, Any, Optional, List +from autoagent.registry import registry +from autoagent import MetaChain +from autoagent.types import Agent, Response +import importlib +import inspect + +# 定义lifespan上下文管理器 +@asynccontextmanager +async def lifespan(app: FastAPI): + # 启动时执行 + await create_agent_endpoints(app) + yield + # 关闭时执行 + # 清理代码（如果需要） + +app = FastAPI(title="MetaChain API", lifespan=lifespan) + +class ToolRequest(BaseModel): + args: Dict[str, Any] + +class AgentRequest(BaseModel): + model: str + query: str + context_variables: Optional[Dict[str, Any]] = {} + +class Message(BaseModel): + role: str + content: str + +class AgentResponse(BaseModel): + result: str + messages: List + agent_name: str +# 为所有注册的tools创建endpoints +@app.on_event("startup") +def create_tool_endpoints(): + for tool_name, tool_func in registry.tools.items(): + # 创建动态的POST endpoint + async def create_tool_endpoint(request: ToolRequest, func=tool_func): + try: + # 检查必需参数 + sig = inspect.signature(func) + required_params = { + name for name, param in sig.parameters.items() + if param.default == inspect.Parameter.empty + } + + # 验证是否提供了所有必需参数 + if not all(param in request.args for param in required_params): + missing = required_params - request.args.keys() + raise HTTPException( + status_code=400, + detail=f"Missing required parameters: {missing}" + ) + + result = func(**request.args) + return {"status": "success", "result": result} + except Exception as e: + raise HTTPException(status_code=400, detail=str(e)) + + # 添加endpoint到FastAPI应用 + endpoint = create_tool_endpoint + endpoint.__name__ = f"tool_{tool_name}" + app.post(f"/tools/{tool_name}")(endpoint) +# 重写agent endpoints创建逻辑 +@app.on_event("startup") +def create_agent_endpoints(): + for agent_name, agent_func in registry.agents.items(): + async def create_agent_endpoint( + request: AgentRequest, + func=agent_func + ) -> AgentResponse: + try: + # 创建agent实例 + agent = func(model=request.model) + + # 创建MetaChain实例 + mc = MetaChain() + + # 构建messages + messages = [ + {"role": "user", "content": request.query} + ] + + # 运行agent + response = mc.run( + agent=agent, + messages=messages, + context_storage=request.context_variables, + debug=True + ) + + return AgentResponse( + result=response.messages[-1]['content'], + messages=response.messages, + agent_name=agent.name + ) + + except Exception as e: + raise HTTPException( + status_code=400, + detail=f"Agent execution failed: {str(e)}" + ) + + endpoint = create_agent_endpoint + endpoint.__name__ = f"agent_{agent_name}" + app.post(f"/agents/{agent_name}/run")(endpoint) + +# 获取所有可用的agents信息 +@app.get("/agents") +async def list_agents(): + return { + name: { + "docstring": info.docstring, + "args": info.args, + "file_path": info.file_path + } + for name, info in registry.agents_info.items() + } + +# 获取特定agent的详细信息 +@app.get("/agents/{agent_name}") +async def get_agent_info(agent_name: str): + if agent_name not in registry.agents_info: + raise HTTPException( + status_code=404, + detail=f"Agent {agent_name} not found" + ) + + info = registry.agents_info[agent_name] + return { + "name": agent_name, + "docstring": info.docstring, + "args": info.args, + "file_path": info.file_path + } + +if __name__ == "__main__": + import uvicorn + uvicorn.run(app, host="0.0.0.0", port=8000) \ No newline at end of file diff --git a/autoagent/tcp_server.py b/autoagent/tcp_server.py new file mode 100644 index 0000000..0d11ba2 --- /dev/null +++ b/autoagent/tcp_server.py @@ -0,0 +1,61 @@ +import socket +import subprocess +import json +import argparse + +parser = argparse.ArgumentParser() +parser.add_argument("--workplace", type=str, default=None) +args = parser.parse_args() + +if __name__ == "__main__": + assert args.workplace is not None, "Workplace is not specified" + server = socket.socket(socket.AF_INET, socket.SOCK_STREAM) + server.bind(("0.0.0.0", 12345)) + server.listen(1) + + print("Listening on port 12345...") + def receive_all(conn, buffer_size=4096): + data = b"" + while True: + part = conn.recv(buffer_size) + data += part + if len(part) < buffer_size: + # 如果接收的数据小于缓冲区大小，可能已经接收完毕 + break + return data.decode() + + while True: + conn, addr = server.accept() + print(f"Connection from {addr}") + while True: + # command = conn.recv(1024).decode() + command = receive_all(conn) + if not command: + break + + # Execute the command + try: + modified_command = f"/bin/bash -c 'source /home/user/micromamba/etc/profile.d/conda.sh && conda activate autogpt && cd /{args.workplace} && {command}'" + process = subprocess.Popen(modified_command, shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True) + output = '' + while True: + line = process.stdout.readline() + if not line and process.poll() is not None: + break + output += line + print(line, end='') + + exit_code = process.wait() + except Exception as e: + exit_code = -1 + output = f"Error running command: {str(e)}" + + # Create a JSON response + response = { + "status": exit_code, + "result": output + } + + # Send the JSON response + conn.send(json.dumps(response).encode()) + conn.close() \ No newline at end of file diff --git a/autoagent/tools/__init__.py b/autoagent/tools/__init__.py new file mode 100644 index 0000000..4453923 --- /dev/null +++ b/autoagent/tools/__init__.py @@ -0,0 +1,67 @@ +# from .code_knowledge import gen_code_tree_structure +# from .execution import execute_command +# from .files import read_file, create_file, write_file, list_files, create_directory +# from .PythonAction import run_python +# from .rag_code import code_rag +# from .tool_retriever import get_api_doc +# from .rag_code_tree import code_tree_rag +# from .inner import case_resolved +# from .code_report import check_tool, check_agent +# from .github_ops import get_current_branch, get_diff, push_changes, submit_pull_request + +# import os +# import importlib +# from autoagent.registry import registry + +# # 获取当前目录下的所有 .py 文件 +# current_dir = os.path.dirname(__file__) +# for file in os.listdir(current_dir): +# if file.endswith('.py') and not file.startswith('__'): +# module_name = file[:-3] +# importlib.import_module(f'autoagent.tools.{module_name}') + +# # 导出所有注册的工具 +# globals().update(registry.tools) + +# __all__ = list(registry.tools.keys()) + +import os +import importlib +from autoagent.registry import registry + +def import_tools_recursively(base_dir: str, base_package: str): + """Recursively import all tools in .py files + + Args: + base_dir: the root directory to start searching + base_package: the base name of the Python package + """ + for root, dirs, files in os.walk(base_dir): + # get the relative path to the base directory + rel_path = os.path.relpath(root, base_dir) + + for file in files: + if file.endswith('.py') and not file.startswith('__'): + # build the module path + if rel_path == '.': + # in the root directory + module_path = f"{base_package}.{file[:-3]}" + else: + # in the subdirectory + package_path = rel_path.replace(os.path.sep, '.') + module_path = f"{base_package}.{package_path}.{file[:-3]}" + + try: + importlib.import_module(module_path) + except Exception as e: + print(f"Warning: Failed to import {module_path}: {e}") + +# get the current directory and import all tools +current_dir = os.path.dirname(__file__) +import_tools_recursively(current_dir, 'autoagent.tools') + +# export all tool creation functions +globals().update(registry.tools) +globals().update(registry.plugin_tools) + +__all__ = list(registry.tools.keys()) + list(registry.plugin_tools.keys()) \ No newline at end of file diff --git a/autoagent/tools/code_search.py b/autoagent/tools/code_search.py new file mode 100644 index 0000000..5dbb487 --- /dev/null +++ b/autoagent/tools/code_search.py @@ -0,0 +1,104 @@ +import requests +from typing import Optional, List, Dict +from autoagent.tools.github_client import GitHubSearcher +from autoagent.registry import register_tool +from constant import GITHUB_AI_TOKEN +import json +@register_tool("search_github_repos") +def search_github_repos(query, limit=5): + """ + Search GitHub public repositories based on a keyword. + + :param query: The query to search for in repository names or descriptions. + :param limit: The total number of repositories to return. + :return: A list of dictionaries containing repository details, limited to the specified number. + """ + repos = [] + per_page = 10 + page = 1 + while len(repos) < limit: + + url = f'https://api.github.com/search/repositories?q={query}&per_page={per_page}&page={page}' + + response = requests.get(url) + + if response.status_code == 200: + items = response.json().get('items', []) + for item in items: + formatted_repo = { + "name": f"{item['owner']['login']}/{item['name']}", + "author": item['owner']['login'], + "description": item['description'], + "link": item['html_url'] + } + repos.append(formatted_repo) + if len(repos) >= limit: + break + + if len(items) < per_page: # Stop if there are no more repos to fetch + break + page += 1 + else: + raise Exception(f"GitHub API request failed with status code {response.status_code}: {response.text}") + + return_str = """ + Here are some of the repositories I found on GitHub: + """ + + for repo in repos: + return_str += f""" + Name: {repo['name']} + Description: {repo['description']} + Link: {repo['link']} + """ + + return return_str +@register_tool("search_github_code") +def search_github_code(repo_owner: str, + repo_name: str, + query: str, + language: Optional[str] = None, + per_page: int = 5, + page: int = 1) -> List[Dict]: + """ + Search GitHub code based on a keyword. + + Args: + repo_owner: The owner of the repository + repo_name: The name of the repository + query: The keyword to search for + language: The programming language to filter by, optional + per_page: The number of results per page, optional + page: The page number, optional + + Returns: + List[Dict]: The search results list + """ + searcher = GitHubSearcher(GITHUB_AI_TOKEN) + results = searcher.search_code(repo_owner, repo_name, query, language, per_page, page) + # print(results) + if 'items' not in results: + return [] + + # Extract useful information + formatted_results = [] + for item in results['items']: + response = requests.get(item['url']) + if response.status_code == 200: + download_url = response.json()['download_url'] + response = requests.get(download_url) + if response.status_code == 200: + content = response.text + else: + content = "" + else: + content = "" + formatted_results.append({ + 'name': item['name'], + 'path': item['path'], + 'url': item['html_url'], + 'repository': item['repository']['full_name'], + 'content_url': item['url'], + 'content': content + }) + return json.dumps(formatted_results, indent=4) diff --git a/autoagent/tools/dummy_tool.py b/autoagent/tools/dummy_tool.py new file mode 100644 index 0000000..f7a7fe4 --- /dev/null +++ b/autoagent/tools/dummy_tool.py @@ -0,0 +1,41 @@ +from autoagent.types import Result, Agent +from typing import Union +from autoagent.registry import register_plugin_tool + +@register_plugin_tool("tool_dummy") # the name of registry should be the same as the name of the tool +def tool_dummy(args1, args2, context_variables)-> Union[str, Agent, Result]: + """ + [This is a dummy tool, it's used for demonstrating the usage of the autoagent, you should write your own tool instead of using this dummy tool, and the following docstring is just a template, you should modify it to fit your own tool.] + The tool is used to ... + + Args: + args1: ... + args2: ... + ... + Returns: + ... + Notes: + The return value can be a string, an agent, or an instance of Result. + 1. The most common return value is a string, but the length of the string should not be too long, and you can save the result to a file if the length is too long. + 2. If the tool is used to transfer the conversation context to an agent, you can return an instance of Agent, like: + ```python + sales_agent = Agent(name="sales_agent", instructions="You are a sales agent.") + def transfer_to_sales_agent(): + return sales_agent + ``` + 3. If there is some complex operations inside the tool, you can return an instance of Result, for example, you should modify the context variables in the result. + ```python + def sales_agent_tool(recommendation: str, context_variables: dict): + ''' + The tool is used to recommend products to the user. + ''' + context_variables["recommendation"] = recommendation + return Result(value="I recommend the product: " + recommendation, agent=user_agent, context_variables=context_variables) + + 4. The value of Result class should be a string, if you want to return a json dictionary, you should convert it to string by json.dumps(result) + ``` + 5. The tools should be created in the python file in the `autoagent/autoagent/tools` folder. + """ + ... # the implementation of the tool + return "..." # the return value of the tool. + diff --git a/autoagent/tools/file_surfer_tool.py b/autoagent/tools/file_surfer_tool.py new file mode 100644 index 0000000..5d09a06 --- /dev/null +++ b/autoagent/tools/file_surfer_tool.py @@ -0,0 +1,346 @@ +from autoagent.environment.markdown_browser import RequestsMarkdownBrowser +from autoagent.environment import LocalEnv +from functools import partial, update_wrapper +from inspect import signature +from typing import Tuple +import time +from autoagent.registry import register_tool, register_plugin_tool +from typing import Union, Optional +from typing import Optional +from autoagent.types import Result +import requests +import mimetypes +import base64 +import uuid +import os +from litellm import completion +import cv2 +import tempfile +from typing import List +from moviepy import * +import time +import base64 +from faster_whisper import WhisperModel +from constant import COMPLETION_MODEL, API_BASE_URL + +def with_env(env: RequestsMarkdownBrowser): + """将env注入到工具函数中的装饰器""" + def decorator(func): + def wrapped(*args, **kwargs): + return func(env=env, *args, **kwargs) + + # 保留原始函数的所有属性 + update_wrapper(wrapped, func) + # 修改signature，移除env参数 + wrapped.__signature__ = signature(func).replace( + parameters=[p for p in signature(func).parameters.values() if p.name != 'env'] + ) + return wrapped + return decorator + +def _get_browser_state(env: RequestsMarkdownBrowser) -> Tuple[str, str]: + """ + Get the current state of the browser, including the header and content. + """ + # print(env.address) + header = f"Address: {env._convert_local_to_docker(env.address)}\n" + # header = f"Address: {env.address}\n" + + if env.page_title is not None: + header += f"Title: {env.page_title}\n" + + current_page = env.viewport_current_page + total_pages = len(env.viewport_pages) + + address = env.address + for i in range(len(env.history) - 2, -1, -1): # Start from the second last + if env.history[i][0] == address: + header += f"You previously visited this page {round(time.time() - env.history[i][1])} seconds ago.\n" + break + prefix = f"Your browser is currently open to the page '{env.page_title}'\n" if env.page_title is not None else "" + + header = prefix + header + header += f"Viewport position: Showing page {current_page+1} of {total_pages}.\n" + return (header, env.viewport) + +@register_tool("open_local_file") +def open_local_file(context_variables, path: str): + """ + Open a local file at a path in the text-based browser and return current viewport content. + + Args: + path: The absolute path of a local file to visit. + """ + env: RequestsMarkdownBrowser = context_variables.get("file_env", None) + assert env is not None, "file_env is not set" + try: + # assert DOCKER_WORKPLACE_NAME in path, f"The path must be a absolute path from `/{DOCKER_WORKPLACE_NAME}/` directory" + # local_path = path.replace('/' + DOCKER_WORKPLACE_NAME, LOCAL_ROOT + f'/{DOCKER_WORKPLACE_NAME}') + # print(local_path) + path = env._convert_docker_to_local(path) + env.open_local_file(path) + header, content = _get_browser_state(env) + final_response = header.strip() + "\n=======================\n" + content + return final_response + except Exception as e: + return f"Error in `open_local_file`: {e}" + +@register_tool("page_up_markdown") +def page_up_markdown(context_variables): + """ + Scroll the viewport UP one page-length in the current file and return the new viewport content. + """ + env: RequestsMarkdownBrowser = context_variables.get("file_env", None) + assert env is not None, "file_env is not set" + try: + env.page_up() + header, content = _get_browser_state(env) + final_response = header.strip() + "\n=======================\n" + content + return final_response + except Exception as e: + return f"Error in `page_up`: {e}" + +@register_tool("page_down_markdown") +def page_down_markdown(context_variables): + """ + Scroll the viewport DOWN one page-length in the current file and return the new viewport content. + """ + env: RequestsMarkdownBrowser = context_variables.get("file_env", None) + assert env is not None, "file_env is not set" + try: + env.page_down() + header, content = _get_browser_state(env) + final_response = header.strip() + "\n=======================\n" + content + return final_response + except Exception as e: + return f"Error in `page_down`: {e}" + +@register_tool("find_on_page_ctrl_f") +def find_on_page_ctrl_f(context_variables, search_string: str): + """ + Scroll the viewport to the first occurrence of the search string. This is equivalent to Ctrl+F. + + Args: + search_string: The string to search for on the page. This search string supports wildcards like '*' + """ + env: RequestsMarkdownBrowser = context_variables.get("file_env", None) + assert env is not None, "file_env is not set" + try: + env.find_on_page(search_string) + header, content = _get_browser_state(env) + final_response = header.strip() + "\n=======================\n" + content + return final_response + except Exception as e: + return f"Error in `find_on_page_ctrl_f`: {e}" + +@register_tool("find_next") +def find_next(context_variables): + """ + Scroll the viewport to next occurrence of the search string. + """ + env: RequestsMarkdownBrowser = context_variables.get("file_env", None) + assert env is not None, "file_env is not set" + try: + env.find_next() + header, content = _get_browser_state(env) + final_response = header.strip() + "\n=======================\n" + content + return final_response + except Exception as e: + return f"Error in `find_next`: {e}" + +def _encode_image(context_variables, image_path: str): + """ + Encode an image to base64. + """ + env: RequestsMarkdownBrowser = context_variables.get("file_env", None) + assert env is not None, "file_env is not set" + if image_path.startswith("http"): + user_agent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/119.0.0.0 Safari/537.36 Edg/119.0.0.0" + request_kwargs = { + "headers": {"User-Agent": user_agent}, + "stream": True, + } + + # Send a HTTP request to the URL + response = requests.get(image_path, **request_kwargs) + response.raise_for_status() + content_type = response.headers.get("content-type", "") + + extension = mimetypes.guess_extension(content_type) + if extension is None: + extension = ".download" + + fname = str(uuid.uuid4()) + extension + download_path = os.path.abspath(os.path.join(env.local_workplace, "downloads", fname)) + + with open(download_path, "wb") as fh: + for chunk in response.iter_content(chunk_size=512): + fh.write(chunk) + + image_path = download_path + elif env.local_workplace in image_path: + image_path = image_path + else: + image_path = env._convert_docker_to_local(image_path) + with open(image_path, "rb") as image_file: + return base64.b64encode(image_file.read()).decode('utf-8') +# @register_tool("visual_question_answering") +# def visual_question_answering(context_variables, image_path: str, question: Optional[str] = None) -> Result: +# """ +# A tool that can answer questions about attached images. +# Args: +# image_path: The path to the image on which to answer the question. This should be a local path to downloaded image. +# question: the question to answer (default: "Please write a detailed caption for this image.") +# """ +# env: RequestsMarkdownBrowser = context_variables.get("file_env", None) +# assert env is not None, "file_env is not set" +# try: + +# if not question: +# question = "Please write a detailed caption for this image." + +# if not isinstance(image_path, str): +# raise Exception("You should provide only one string as argument to this tool!") + +# base64_image = _encode_image(context_variables, image_path) + +# ret_str = question + +# msg = [{"role": "user", "content": [ +# {"type": "text", "text": question}, +# {"type": "image_url", "image_url": {"url": f"data:image/png;base64,{base64_image}"}} +# ]}] +# res = completion(model="gpt-4o-2024-08-06", messages=msg) +# ret_str = res.choices[0].message.content +# return Result( +# value=ret_str, +# # image=base64_image +# ) +# except Exception as e: +# return Result( +# value=f"Error in `visual_question_answering`: {e}", +# ) + +@register_tool("visual_question_answering") +@register_plugin_tool("visual_question_answering") +def visual_question_answering(context_variables, file_path: str, question: Optional[str] = None) -> Result: + """ + This tool is used to answer questions about attached images or videos. + Args: + file_path: File path of the image or video. + question: The question to answer (default: "Please describe the image or video content.") + """ + env: Union[RequestsMarkdownBrowser, LocalEnv] = context_variables.get("file_env", LocalEnv()) + if env is None: + env = LocalEnv() + + try: + if not isinstance(file_path, str): + raise Exception("File path must be a string!") + + # 判断文件类型 + file_extension = os.path.splitext(file_path)[1].lower() + video_extensions = {'.mp4', '.avi', '.mov', '.mkv', '.webm'} + + if file_extension in video_extensions: + # 视频处理逻辑 + if not question: + question = "Please describe the video content." + + # 1. 提取关键帧 + local_file_path = env._convert_docker_to_local(file_path) + base64Frames, audio_path = process_video(local_file_path) + if audio_path is not None: + audio_text = process_audio(audio_path) + else: + audio_text = "No audio found in the video." + + messages=[ + {"role": "system", "content":"""Use the video and transcription to answer the provided question."""}, + {"role": "user", "content": [ + {"type": "text", "text": "These are the frames from the video."}, + *map(lambda x: {"type": "image_url", + "image_url": {"url": f'data:image/png;base64,{x}', "detail": "low"}}, base64Frames), + {"type": "text", "text": f"The audio transcription is: {audio_text}\nQuestion: {question}"}, + ], + } + ] + final_res = completion(model=COMPLETION_MODEL, messages=messages, base_url=API_BASE_URL) + return Result(value=final_res.choices[0].message.content) + + else: + # 原有的图片处理逻辑 + if not question: + question = "Please describe the image content." + + base64_image = _encode_image(context_variables, file_path) + msg = [{"role": "user", "content": [ + {"type": "text", "text": question}, + {"type": "image_url", "image_url": {"url": f"data:image/png;base64,{base64_image}"}} + ]}] + res = completion(model=COMPLETION_MODEL, messages=msg, base_url=API_BASE_URL) + return Result(value=res.choices[0].message.content) + + except Exception as e: + return Result(value=f"Error in `visual_question_answering`: {e}") +def process_audio(audio_path): + model = WhisperModel("large-v3-turbo") + print(f"Processing batch: {audio_path}") + + segments, info = model.transcribe(audio_path) + transcribed_text = "" + for segment in segments: + transcribed_text += "[%.2fs -> %.2fs] %s\n" % (segment.start, segment.end, segment.text) + + return transcribed_text +def process_video(video_path, seconds_per_frame=2): + base64Frames = [] + base_video_path, _ = os.path.splitext(video_path) + + video = cv2.VideoCapture(video_path) + total_frames = int(video.get(cv2.CAP_PROP_FRAME_COUNT)) + fps = video.get(cv2.CAP_PROP_FPS) + frames_to_skip = int(fps * seconds_per_frame) + curr_frame=0 + + # Loop through the video and extract frames at specified sampling rate + while curr_frame < total_frames - 1: + video.set(cv2.CAP_PROP_POS_FRAMES, curr_frame) + success, frame = video.read() + if not success: + break + _, buffer = cv2.imencode(".png", frame) + base64Frames.append(base64.b64encode(buffer).decode("utf-8")) + curr_frame += frames_to_skip + video.release() + + # Extract audio from video + + clip = VideoFileClip(video_path) + if clip.audio is not None: + audio_path = f"{base_video_path}.mp3" + clip.audio.write_audiofile(audio_path, bitrate="32k") + clip.audio.close() + clip.close() + else: + audio_path = None + clip.close() + + print(f"Extracted {len(base64Frames)} frames") + print(f"Extracted audio to {audio_path}") + return base64Frames, audio_path + +if __name__ == "__main__": + local_root = os.getcwd() + workplace_name = 'workplace_gaia_meta' + env = RequestsMarkdownBrowser(viewport_size=1024 * 5, local_root=local_root, workplace_name=workplace_name, downloads_folder=os.path.join(local_root, workplace_name, "downloads")) + # print("Open file", "~"*100) + # print(open_local_file(env, f"/{workplace_name}/downloads/2207.01510v1.pdf")) + # print("Page down", "~"*100) + # print(page_down_markdown(env)) + # print("Find on page", "~"*100) + # print(find_on_page_ctrl_f(env, "Chain-of-Thought")) + # print("Find next", "~"*100) + # print(find_next(env)) + print(visual_question_answering(context_variables = {"file_env": env}, file_path = "//workplace_gaia_meta/autoagent/downloaded_video.mp4", question="What is the highest number of bird species to be on camera simultaneously?").value) + # print(visual_question_answering("/workplace_meta/downloads/workflow.png", "What is the main idea of this paper?").image) \ No newline at end of file diff --git a/autoagent/tools/github_client.py b/autoagent/tools/github_client.py new file mode 100644 index 0000000..39688ce --- /dev/null +++ b/autoagent/tools/github_client.py @@ -0,0 +1,256 @@ +import os +import requests +import json +from typing import Optional, Dict, List +import time +class GitHubClient: + """GitHub operation client""" + + def __init__(self, token: Optional[str] = None): + """ + Initialize the GitHub client + + Args: + token: GitHub Personal Access Token, if None, try to get from environment variable + """ + self.token = token or os.getenv('GITHUB_AI_TOKEN') + if not self.token: + raise ValueError("GitHub Token is required, please provide it via the token parameter or set the GITHUB_AI_TOKEN environment variable.") + + self.session = requests.Session() + self.session.headers.update({ + 'Authorization': f'token {self.token}', + 'Accept': 'application/vnd.github.v3+json' + }) + self.api_base = 'https://api.github.com' + + def check_auth(self) -> dict: + """ + Check if the token is valid + """ + try: + response = self.session.get(f'{self.api_base}/user') + response.raise_for_status() + return {'status': 0, 'message': 'Authentication successful', 'user': response.json()} + except Exception as e: + return {'status': -1, 'message': f'Authentication failed: {str(e)}'} + + def create_pull_request(self, repo: str, title: str, body: str, head: str, base: str) -> dict: + """ + Create a Pull Request + + Args: + repo: The full name of the repository (e.g., 'owner/repo') + title: The PR title + body: The PR description + head: The source branch + base: The target branch + """ + try: + url = f'{self.api_base}/repos/{repo}/pulls' + data = { + 'title': title, + 'body': body, + 'head': head, + 'base': base + } + response = self.session.post(url, json=data) + response.raise_for_status() + pr_data = response.json() + return { + 'status': 0, + 'message': f'PR created successfully: {pr_data["html_url"]}', + 'pr_url': pr_data['html_url'] + } + except Exception as e: + return {'status': -1, 'message': f'PR creation failed: {str(e)}'} + + + +class GitHubSearcher: + def __init__(self, token: Optional[str] = None): + """ + Initialize the GitHub searcher + + Args: + token: GitHub Personal Access Token, optional + """ + self.session = requests.Session() + if token: + self.session.headers.update({ + 'Authorization': f'token {token}', + 'Accept': 'application/vnd.github.v3+json' + }) + self.session.headers.update({ + 'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36' + }) + + # def search_code(self, + # repo_owner: str, + # repo_name: str, + # query: str, + # language: Optional[str] = None, + # per_page: int = 1) -> Dict: + # """ + # Search code in a specific repository + + # Args: + # repo_owner: The owner of the repository + # repo_name: The name of the repository + # query: The search keyword + # language: The programming language filter, optional + # per_page: The number of results per page + + # Returns: + # dict: The search results + # """ + # # Modify the search URL + # base_url = "https://api.github.com/search/code" # Modify here + + # # Build the query parameters + # q = f"repo:{repo_owner}/{repo_name} {query}" + # if language: + # q += f" language:{language}" + + # params = { + # 'q': q, + # 'per_page': per_page # add this parameter + # } + + # try: + # response = self.session.get(base_url, params=params) + # response.raise_for_status() # Check if the request is successful + + # # Handle rate limiting + # if 'X-RateLimit-Remaining' in response.headers: + # remaining = int(response.headers['X-RateLimit-Remaining']) + # if remaining < 10: # If the remaining requests are less, pause for a while + # reset_time = int(response.headers['X-RateLimit-Reset']) + # sleep_time = reset_time - time.time() + # if sleep_time > 0: + # time.sleep(sleep_time) + + # return response.json() + + # except requests.exceptions.RequestException as e: + # return { + # 'status': 'error', + # 'message': f"Request failed: {str(e)}", + # 'items': [] + # } + + # def get_file_content(self, file_url: str) -> str: + # """ + # Get the content of a file + + # Args: + # file_url: The URL of the file + + # Returns: + # str: The content of the file + # """ + # try: + # response = self.session.get(file_url) + # response.raise_for_status() + # return response.json()['content'] + + # except requests.exceptions.RequestException as e: + # return f"Failed to get file content: {str(e)}" + def search_code(self, + repo_owner: str, + repo_name: str, + query: str, + language: Optional[str] = None, + per_page: int = 5, + page: int = 1) -> Dict: + """搜索代码""" + base_url = "https://api.github.com/search/code" + + # 构建查询 + q = f"repo:{repo_owner}/{repo_name} {query}" + if language: + q += f" language:{language}" + + params = { + 'q': q, + 'per_page': min(per_page, 100), # 确保不超过最大限制 + 'page': page + } + + try: + response = self.session.get(base_url, params=params) + response.raise_for_status() + + # 处理速率限制 + self._handle_rate_limit(response.headers) + + return response.json() + + except requests.exceptions.RequestException as e: + return { + 'status': 'error', + 'message': f"Request failed: {str(e)}", + 'items': [] + } + + def get_contents_batch(self, items: List[Dict]) -> List[Dict]: + """批量获取文件内容""" + results = [] + with concurrent.futures.ThreadPoolExecutor(max_workers=10) as executor: + future_to_item = { + executor.submit(self._get_single_content, item): item + for item in items + } + + for future in concurrent.futures.as_completed(future_to_item): + try: + result = future.result() + results.append(result) + except Exception as e: + print(f"Error getting content: {str(e)}") + + return results + + def _get_single_content(self, item: Dict) -> Dict: + """获取单个文件的内容""" + try: + response = self.session.get(item['url']) + response.raise_for_status() + self._handle_rate_limit(response.headers) + + file_data = response.json() + if 'download_url' in file_data: + content_response = self.session.get(file_data['download_url']) + content_response.raise_for_status() + content = content_response.text + else: + content = file_data.get('content', '') + if content: + import base64 + content = base64.b64decode(content).decode('utf-8') + + return { + 'name': item['name'], + 'path': item['path'], + 'url': item['html_url'], + 'repository': item['repository']['full_name'], + 'content': content + } + except Exception as e: + return { + 'name': item['name'], + 'path': item['path'], + 'url': item['html_url'], + 'repository': item['repository']['full_name'], + 'content': f"Error: {str(e)}" + } + + def _handle_rate_limit(self, headers: Dict): + """处理 API 速率限制""" + if 'X-RateLimit-Remaining' in headers: + remaining = int(headers['X-RateLimit-Remaining']) + if remaining < 10: + reset_time = int(headers['X-RateLimit-Reset']) + sleep_time = reset_time - time.time() + if sleep_time > 0: + time.sleep(min(sleep_time, 5)) # 最多等待5秒 \ No newline at end of file diff --git a/autoagent/tools/github_ops.py b/autoagent/tools/github_ops.py new file mode 100644 index 0000000..5d5bc72 --- /dev/null +++ b/autoagent/tools/github_ops.py @@ -0,0 +1,153 @@ +# from autoagent.util import run_command_in_container +from autoagent.environment import DockerEnv, LocalEnv +from constant import GITHUB_AI_TOKEN +from autoagent.tools.github_client import GitHubClient +import json +from autoagent.registry import register_tool +from typing import Union +@register_tool("get_current_branch") +def get_current_branch(context_variables): + f""" + Get the current branch of the 'autoagent'. + """ + env: Union[DockerEnv, LocalEnv] = context_variables.get("code_env", LocalEnv()) + branch_command = f"cd {env.docker_workplace}/autoagent && git branch --show-current" + result = env.run_command(branch_command) + if result['status'] == 0: + return result['result'].strip() + else: + return f"Failed to get the current branch. Error: {result['result'].strip()}" + +@register_tool("get_diff") +def get_diff(context_variables): + f""" + Get the diff of the 'autoagent'. + """ + env: Union[DockerEnv, LocalEnv] = context_variables.get("code_env", LocalEnv()) + diff_command = f"cd {env.docker_workplace}/autoagent && git add -N . && git diff" + result = env.run_command(diff_command) + if result['status'] == 0: + return result['result'].strip() + else: + return f"Failed to get the diff. Error: {result['result'].strip()}" + +@register_tool("stage_files") +def stage_files(context_variables, file_paths=None): + """ + Stage the specified file changes + + Args: + file_paths (list): The file paths to stage, if None, add all changes to the staging area + + Returns: + dict: The operation result + """ + env: Union[DockerEnv, LocalEnv] = context_variables.get("code_env", LocalEnv()) + if file_paths is None: + # add all changes to the staging area + command = f"cd {env.docker_workplace}/autoagent && git add ." + else: + # add specified files to the staging area + files = ' '.join(file_paths) + command = f"cd {env.docker_workplace}/autoagent && git add {files}" + + result = env.run_command(command) + return result + +@register_tool("push_changes") +def push_changes(context_variables, commit_message, file_paths=None): + """ + Push the selected changes to the remote repository + + Args: + commit_message (str): The commit message + file_paths (list): The file paths to commit, if None, commit all changes + + Returns: + dict: The push result + """ + # stage the files + # if file_paths: + env: Union[DockerEnv, LocalEnv] = context_variables.get("code_env", LocalEnv()) + stage_result = stage_files(env, file_paths) + if stage_result['status'] != 0: + return json.dumps({'status': 'error', 'message': f"Failed to stage files: {stage_result['result']}"}, indent=4) + + commands = [ + f"cd {env.docker_workplace}/autoagent", + f'git commit -m "{commit_message}"', + "git push origin $(git branch --show-current)" + ] + + command = " && ".join(commands) + result = env.run_command(command) + + if result['status'] == 0: + return f"push success. {result['result']}" + else: + return f"push failed. {result['result']}" + +@register_tool("submit_pull_request") +def submit_pull_request(title: str, body: str, target_branch: str): + """ + Submit a Pull Request + + Args: + title: PR title + body: PR description + target_branch: target branch + """ + # initialize GitHub client + github = GitHubClient(GITHUB_AI_TOKEN) + + # check authentication + auth_result = github.check_auth() + if auth_result['status'] != 0: + return auth_result + + # create a pull request + pr_result = github.create_pull_request( + repo="tjb-tech/autoagent", + title=title, + body=body, + head=get_current_branch(), + base=target_branch + ) + if pr_result['status'] == 0: + return f"PR created successfully: {json.dumps(pr_result, indent=4)}" + else: + return f"PR creation failed: {json.dumps(pr_result, indent=4)}" + +# def create_pull_request(title, body, target_branch): +# """ +# Create a Pull Request to the target branch + +# Args: +# title (str): The title of the PR +# body (str): The description content of the PR +# target_branch (str): The target branch name + +# Returns: +# dict: PR creation result +# """ + +# # use gh to create a PR. make sure the gh cli is installed in the container and the github token is set +# pr_command = f"""cd /{DOCKER_WORKPLACE_NAME}/autoagent && \ +# gh pr create \ +# --title "{title}" \ +# --body "{body}" \ +# --base {target_branch} \ +# --head $(git branch --show-current)""" + +# result = run_command_in_container(pr_command) + +# if result['status'] == 0: +# return f"PR created successfully: {result['result']}" +# else: +# return f"PR creation failed: {result['result']}" +if __name__ == "__main__": + from rich import print + print("Current branch: " + get_current_branch()) + print("Diff: " + get_diff()) + print(push_changes(commit_message="test")) + print(submit_pull_request(title="test", body="test", target_branch="test_pull_1107")) \ No newline at end of file diff --git a/autoagent/tools/inner.py b/autoagent/tools/inner.py new file mode 100644 index 0000000..569df2f --- /dev/null +++ b/autoagent/tools/inner.py @@ -0,0 +1,27 @@ +from autoagent.registry import register_tool + +@register_tool("case_resolved") +def case_resolved(result: str): + """ + Use this function when the case is resolved and no further actions are needed. Please encapsulate your final answer (answer ONLY) within and . + + Args: + result: The final result of the case resolution following the instructions. + + Example: case_resolved(`The answer to the question is 42 `) + """ + return f"Case resolved. No further actions are needed. The result of the case resolution is: {result}" + +@register_tool("case_not_resolved") +def case_not_resolved(failure_reason: str): + """ + Use this function when the case is not resolved when all agents have tried their best. + [IMPORTANT] Please do not use this function unless all of you have tried your best. + + Args: + failure_reason: The reason why the case is not resolved. + """ + return f"Case not resolved. No further actions are needed. The reason is: {failure_reason}" + +if __name__ == "__main__": + print(case_resolved()) \ No newline at end of file diff --git a/autoagent/tools/md_obs.py b/autoagent/tools/md_obs.py new file mode 100644 index 0000000..58e0051 --- /dev/null +++ b/autoagent/tools/md_obs.py @@ -0,0 +1,208 @@ +from browsergym.utils.obs import _process_bid, IGNORED_AXTREE_ROLES, IGNORED_AXTREE_PROPERTIES + +def flatten_md_axtree_to_str( + AX_tree, + extra_properties: dict = None, + with_visible: bool = False, + with_clickable: bool = False, + with_center_coords: bool = False, + with_bounding_box_coords: bool = False, + with_som: bool = False, + skip_generic: bool = True, + filter_visible_only: bool = False, + filter_with_bid_only: bool = False, + filter_som_only: bool = False, + coord_decimals: int = 0, + ignored_roles=IGNORED_AXTREE_ROLES, + ignored_properties=IGNORED_AXTREE_PROPERTIES, + remove_redundant_static_text: bool = True, + hide_bid_if_invisible: bool = False, + hide_all_children: bool = False, + hide_all_bids: bool = False, +) -> str: + """Formats the accessibility tree into a string text""" + node_id_to_idx = {} + for idx, node in enumerate(AX_tree["nodes"]): + node_id_to_idx[node["nodeId"]] = idx + # print("******************************* AX_tree:", AX_tree) + + def dfs(node_idx: int, depth: int, parent_node_filtered: bool, parent_node_name: str) -> str: + tree_str = "" + node = AX_tree["nodes"][node_idx] + # indent = "\t" * depth + indent = "" + skip_node = False # node will not be printed, with no effect on children nodes + filter_node = False # node will not be printed, possibly along with its children nodes + node_role = node["role"]["value"] + node_name = "" + + if node_role in ignored_roles: + skip_node = True + pass + elif "name" not in node: + skip_node = True + pass + else: + node_name = node["name"]["value"] + if "value" in node and "value" in node["value"]: + node_value = node["value"]["value"] + else: + node_value = None + + # extract bid + bid = node.get("browsergym_id", None) + + # extract node attributes + attributes = [] + for property in node.get("properties", []): + if not "value" in property: + continue + if not "value" in property["value"]: + continue + + prop_name = property["name"] + prop_value = property["value"]["value"] + + if prop_name in ignored_properties: + continue + elif prop_name in ("required", "focused", "atomic"): + if prop_value: + attributes.append(prop_name) + else: + attributes.append(f"{prop_name}={repr(prop_value)}") + + if skip_generic and node_role == "generic" and not attributes: + skip_node = True + + if hide_all_children and parent_node_filtered: + skip_node = True + + if node_role == "StaticText": + if parent_node_filtered: + skip_node = True + elif remove_redundant_static_text and node_name in parent_node_name: + skip_node = True + else: + filter_node, extra_attributes_to_print = _process_bid( + bid, + extra_properties=extra_properties, + with_visible=with_visible, + with_clickable=with_clickable, + with_center_coords=with_center_coords, + with_bounding_box_coords=with_bounding_box_coords, + with_som=with_som, + filter_visible_only=filter_visible_only, + filter_with_bid_only=filter_with_bid_only, + filter_som_only=filter_som_only, + coord_decimals=coord_decimals, + ) + + # if either is True, skip the node + skip_node = skip_node or filter_node + + # insert extra attributes before regular attributes + attributes = extra_attributes_to_print + attributes + + # actually print the node string + if not skip_node: + if node_role == "generic" and not node_name: + node_str = f"{node_role}" + else: + node_str = f"{node_name.strip()}" + + if not ( + hide_all_bids + or bid is None + or ( + hide_bid_if_invisible + and extra_properties.get(bid, {}).get("visibility", 0) < 0.5 + ) + ): + node_str = node_str + + if node_value is not None: + node_str += f' value={repr(node["value"]["value"])}' + + if attributes: + node_str += ", ".join([""] + attributes) + + tree_str += f"{indent}{node_str}" + + for child_node_id in node["childIds"]: + if child_node_id not in node_id_to_idx or child_node_id == node["nodeId"]: + continue + # mark this to save some tokens + child_depth = depth if skip_node else (depth + 1) + child_str = dfs( + node_id_to_idx[child_node_id], + child_depth, + parent_node_filtered=filter_node, + parent_node_name=node_name, + ) + if child_str: + if tree_str: + tree_str += "\n" + tree_str += child_str + + return tree_str + + tree_str = dfs(0, 1, False, "") + return tree_str + +# def flatten_md_axtree_to_str( +# AX_tree, +# extra_properties: dict = None, +# with_visible: bool = False, +# with_clickable: bool = False, +# with_center_coords: bool = False, +# with_bounding_box_coords: bool = False, +# with_som: bool = False, +# skip_generic: bool = True, +# filter_visible_only: bool = False, +# filter_with_bid_only: bool = False, +# filter_som_only: bool = False, +# coord_decimals: int = 0, +# ignored_roles=IGNORED_AXTREE_ROLES, +# ignored_properties=IGNORED_AXTREE_PROPERTIES, +# remove_redundant_static_text: bool = True, +# hide_bid_if_invisible: bool = False, +# hide_all_children: bool = False, +# hide_all_bids: bool = False, +# ) -> str: +# """Formats the accessibility tree into a markdown string""" +# if not AX_tree or "nodes" not in AX_tree or not AX_tree["nodes"]: +# return "No content available" + +# node_id_to_idx = {} +# for idx, node in enumerate(AX_tree["nodes"]): +# node_id_to_idx[node["nodeId"]] = idx + +# def dfs(node_idx: int, depth: int, parent_node_filtered: bool, parent_node_name: str) -> str: +# tree_str = "" +# node = AX_tree["nodes"][node_idx] + +# # 处理节点内容 +# if "name" in node and "value" in node["name"]: +# content = node["name"]["value"].strip() +# if content: +# # 移除多余的空白字符 +# content = ' '.join(content.split()) +# # 添加到输出 +# tree_str += content + "\n" + +# # 递归处理子节点 +# for child_node_id in node.get("childIds", []): +# if child_node_id in node_id_to_idx and child_node_id != node["nodeId"]: +# child_str = dfs( +# node_id_to_idx[child_node_id], +# depth + 1, +# parent_node_filtered=False, +# parent_node_name=node.get("name", {}).get("value", "") +# ) +# if child_str: +# tree_str += child_str + +# return tree_str + +# tree_str = dfs(0, 0, False, "") +# return tree_str.strip() \ No newline at end of file diff --git a/autoagent/tools/meta/edit_agents.py b/autoagent/tools/meta/edit_agents.py new file mode 100644 index 0000000..c612d93 --- /dev/null +++ b/autoagent/tools/meta/edit_agents.py @@ -0,0 +1,436 @@ +from autoagent.registry import registry +from autoagent.environment import LocalEnv, DockerEnv, DockerConfig +from typing import Union +from autoagent.tools.terminal_tools import ( + create_file, + create_directory, + execute_command, + run_python, + print_stream, + process_terminal_response + ) +from autoagent.registry import register_tool +import json +from autoagent.tools.meta.edit_tools import get_metachain_path +from string import Formatter +from pydantic import BaseModel +import subprocess +import sys +import shlex +from datetime import datetime +@register_tool("list_agents") +def list_agents(context_variables): + """ + List all plugin agents in the MetaChain. + Returns: + A list of information of all plugin agents including name, args, docstring, body, return_type, file_path. + """ + env: Union[LocalEnv, DockerEnv] = context_variables.get("code_env", LocalEnv()) + try: + path = get_metachain_path(env) + except Exception as e: + return "[ERROR] Failed to list agents. Error: " + str(e) + python_code = '"from autoagent.registry import registry; import json; print(\\"AGENT_LIST_START\\"); print(json.dumps(registry.display_plugin_agents_info, indent=4)); print(\\"AGENT_LIST_END\\")"' + list_agents_cmd = f"cd {path} && DEFAULT_LOG=False python -c {python_code}" + result = env.run_command(list_agents_cmd) + if result['status'] != 0: + return "[ERROR] Failed to list agents. Error: " + result['result'] + try: + output = result['result'] + start_marker = "AGENT_LIST_START" + end_marker = "AGENT_LIST_END" + start_idx = output.find(start_marker) + len(start_marker) + end_idx = output.find(end_marker) + + if start_idx == -1 or end_idx == -1: + return "[ERROR] Failed to parse agent list: markers not found" + + json_str = output[start_idx:end_idx].strip() + return json_str + except Exception as e: + return f"[ERROR] Failed to process output: {str(e)}" + + +@register_tool("delete_agent") +def delete_agent(agent_name: str, context_variables): + """ + Delete a plugin agent. + Args: + agent_name: The name of the agent to be deleted. + Returns: + A string representation of the result of the agent deletion. + """ + env: Union[LocalEnv, DockerEnv] = context_variables.get("code_env", LocalEnv()) + try: + agent_list = list_agents(context_variables) + if agent_list.startswith("[ERROR]"): + return "[ERROR] Failed to list agents. Error: " + agent_list + agent_dict = json.loads(agent_list) + if agent_name in agent_dict.keys(): + agent_info = agent_dict[agent_name] + else: + return "[ERROR] The agent " + agent_name + " does not exist." + except Exception as e: + return "[ERROR] Before deleting a agent, you should list all agents first. But the following error occurred: " + str(e) + + agent_path = agent_info['file_path'] + try: + result = env.run_command(f"rm {agent_path}") + if result['status'] != 0: + return f"[ERROR] Failed to delete agent: `{agent_name}`. Error: " + result['result'] + return f"[SUCCESS] Successfully deleted agent: `{agent_name}`." + except Exception as e: + return f"[ERROR] Failed to delete agent: `{agent_name}`. Error: " + str(e) + +@register_tool("run_agent") +@process_terminal_response +def run_agent(agent_name: str, query: str, ctx_vars: dict, context_variables, model: str = "claude-3-5-sonnet-20241022"): + """ + Run a plugin agent. + Args: + agent_name: The name of the agent. + model: The model to be used for the agent. Supported models: claude-3-5-sonnet-20241022. + query: The query to be used for the agent. + ctx_vars: The global context variables to be used for the agent. It is a dictionary with the key as the variable name and the value as the variable value. + Returns: + A string representation of the result of the agent run. + """ + if model not in ["claude-3-5-sonnet-20241022"]: + return "[ERROR] The model " + model + " is not supported. Supported models: claude-3-5-sonnet-20241022." + env: Union[LocalEnv, DockerEnv] = context_variables.get("code_env", LocalEnv()) + try: + path = get_metachain_path(env) + except Exception as e: + return "[ERROR] Failed to get the path of the MetaChain. Error: " + str(e) + + try: + agent_list = list_agents(context_variables) + if agent_list.startswith("[ERROR]"): + return "[ERROR] Failed to list agents. Error: " + agent_list + agent_dict = json.loads(agent_list) + if agent_name in agent_dict.keys(): + agent_info = agent_dict[agent_name] + agent_func = agent_info['func_name'] + else: + return "[ERROR] The agent " + agent_name + " does not exist." + except Exception as e: + return "[ERROR] Before running a agent, you should list all agents first. But the following error occurred: " + str(e) + if isinstance(ctx_vars, dict) is False: + try: + ctx_vars = json.loads(ctx_vars) + except Exception as e: + return "[ERROR] The context variables are not a valid JSON object. Error: " + str(e) + + ctx_vars_str = "" + for key, value in ctx_vars.items(): + ctx_vars_str += f"{key}={value} " + try: + # query = shlex.quote(query) + # run_cmd = f'cd {path} && DEFAULT_LOG=False mc agent --model={model} --agent_func={agent_func} --query={query} {ctx_vars_str}' + query = shlex.quote(query) + shell_content = f"""#!/bin/bash +cd {path} +DEFAULT_LOG=False mc agent --model={model} --agent_func={agent_func} --query={query} {ctx_vars_str} +""" + timestamp = datetime.now().strftime("%Y%m%d_%H%M%S") + create_directory(f"{path}/tmp_shell", context_variables) + create_file(f"{path}/tmp_shell/run_agent_{timestamp}.sh", shell_content, context_variables) + run_cmd = f"cd {path} && chmod +x tmp_shell/run_agent_{timestamp}.sh && ./tmp_shell/run_agent_{timestamp}.sh" + result = env.run_command(run_cmd, print_stream) + # if result['status'] != 0: + # return f"[ERROR] Failed to run agent: `{agent_func}`. Error: " + result['result'] + # return f"[SUCCESS] Successfully run agent: `{agent_func}`. The result is: \n{result['result']}" + return result + except Exception as e: + return "[ERROR] Failed to run the agent. Error: " + str(e) + +def has_format_keys(s): + formatter = Formatter() + return any(tuple_item[1] is not None for tuple_item in formatter.parse(s)) +def extract_format_keys(s): + formatter = Formatter() + ret_list = [] + for tuple_item in formatter.parse(s): + if tuple_item[1] is not None and tuple_item[1] not in ret_list: + ret_list.append(tuple_item[1]) + return ret_list +@register_tool("create_agent") +def create_agent(agent_name: str, agent_description: str, agent_tools: list[str], agent_instructions: str, context_variables): + """ + Use this tool to create a new agent or modify an existing agent. + + Args: + agent_name: The name of the agent. + agent_description: The description of the agent. + agent_tools: The tools of the agent. The tools MUST be included in the list of given tools. + agent_instructions: The system instructions of the agent, which tells the agent about the responsibility of the agent, the tools it can use and other important information. It could be a pure string or a string with the format of {global_keys}, where the global keys are the keys of the variables that are given to the agent. + + Returns: + A string representation of the result of the agent creation or modification. + """ + tools_str = "" + code_env: Union[LocalEnv, DockerEnv] = context_variables.get("code_env", LocalEnv()) + try: + path = get_metachain_path(code_env) + except Exception as e: + return "[ERROR] Failed to list agents. Error: " + str(e) + + agents_dir = path + "/autoagent/agents" + + for tool in agent_tools: + tools_str += f"from autoagent.tools import {tool}\n" + agent_func = f"get_{agent_name.lower().replace(' ', '_')}" + if has_format_keys(agent_instructions): + format_keys = extract_format_keys(agent_instructions) + format_keys_values = [] + for fk in format_keys: + format_keys_values.append(f"{fk}=context_variables.get('{fk}', '')") + format_keys_values_str = ", ".join(format_keys_values) + instructions_str = f"""\ +def instructions(context_variables): + return {repr(agent_instructions)}.format({format_keys_values_str}) +""" + else: + instructions_str = f"""instructions = {repr(agent_instructions)}""" + tool_list = "[{}]".format(', '.join(f'{tool}' for tool in agent_tools)) + + create_codes = f"""\ +from autoagent.types import Agent +{tools_str} +from autoagent.registry import register_plugin_agent + +@register_plugin_agent(name="{agent_name}", func_name="{agent_func}") +def {agent_func}(model: str): + ''' + {agent_description} + ''' + {instructions_str} + return Agent( + name="{agent_name}", + model=model, + instructions=instructions, + functions={tool_list} + ) + +""" + # print(create_codes) + # with open(f"autoagent/agents/{agent_name.lower().replace(' ', '_')}.py", "w", encoding="utf-8") as f: + # f.write(create_codes) + try: + msg = create_file(agents_dir + "/" + agent_name.lower().replace(' ', '_') + ".py", create_codes, context_variables) + if msg.startswith("Error creating file:"): + return "[ERROR] Failed to create agent. Error: " + msg + result = code_env.run_command('cd {} && python autoagent/agents/{}.py'.format(path, agent_name.lower().replace(' ', '_'))) + if result['status'] != 0: + return "[ERROR] Failed to create agent. Error: " + result['result'] + return "Successfully created agent: " + agent_name + " in " + agents_dir + "/" + agent_name.lower().replace(' ', '_') + ".py" + except Exception as e: + return "[ERROR] Failed to create agent. Error: " + str(e) + +class SubAgent(BaseModel): + name: str + agent_input: str + agent_output: str +@register_tool("create_orchestrator_agent") +def create_orchestrator_agent(agent_name: str, agent_description: str, sub_agents: list[SubAgent], agent_instructions: str, context_variables): + """ + Use this tool to create a orchestrator agent for the given sub-agents. You MUST use this tool when you need to create TWO or MORE agents and regard them as a whole to complete a task. + + Args: + agent_name: The name of the orchestrator agent for the given sub-agents. + agent_description: The description of the orchestrator agent. + sub_agents: The list of sub-agents. Each sub-agent contains the name of the sub-agent, the input of the sub-agent and the output of the sub-agent. + agent_instructions: The system instructions of the orchestrator agent, which tells the agent about the responsibility of the agent (orchestrate the workflow of the given sub-agents), the given sub-agents and other important information. It could be a pure string or a string with the format of {global_keys}, where the global keys are the keys of the variables that are given to the agent. + + Returns: + A string representation of the result of the agent creation or modification. + """ + + code_env: Union[LocalEnv, DockerEnv] = context_variables.get("code_env", LocalEnv()) + try: + path = get_metachain_path(code_env) + except Exception as e: + return "[ERROR] Failed to list agents. Error: " + str(e) + + agents_dir = path + "/autoagent/agents" + agent_list = list_agents(context_variables) + if agent_list.startswith("[ERROR]"): + return "Failed to list agents. Error: " + agent_list + agent_dict = json.loads(agent_list) + sub_agent_info = [agent_dict[sub_agent["name"]] for sub_agent in sub_agents] + import_agent_str = "" + for ainfo in sub_agent_info: + import_agent_str += f""" + from autoagent.agents import {ainfo['func_name']} +""" + if has_format_keys(agent_instructions): + format_keys = extract_format_keys(agent_instructions) + format_keys_values = [] + for fk in format_keys: + format_keys_values.append(f"{fk}=context_variables.get('{fk}', '')") + format_keys_values_str = ", ".join(format_keys_values) + instructions_str = f"""\ +def instructions(context_variables): + return {repr(agent_instructions)}.format({format_keys_values_str}) +""" + else: + instructions_str = f"""instructions = {repr(agent_instructions)}""" + orchestrator_agent_def = f""" + {agent_name.lower().replace(' ', '_')} = Agent( + name="{agent_name}", + model=model, + instructions=instructions, + ) +""" + sub_agent_funcs = [ainfo['func_name'] for ainfo in sub_agent_info] + get_sub_agents = "" + transfer_sub_agent_func = "" + transfer_back_to_orchestrator_func = "" + transfer_funcs_str = [] + for sub_agent_func, sub_agent in zip(sub_agent_funcs, sub_agents): + get_sub_agents += f""" + {sub_agent_func.replace('get_', '')}: Agent = {sub_agent_func}(model) + {sub_agent_func.replace('get_', '')}.tool_choice = "required" +""" + transfer_sub_agent_func += f""" + def transfer_to_{sub_agent_func.replace('get_', '')}({sub_agent["agent_input"]}: str): + ''' + Use this tool to transfer the request to the `{sub_agent_func.replace('get_', '')}` agent. + + Args: + {sub_agent["agent_input"]}: the request to be transferred to the `{sub_agent_func.replace('get_', '')}` agent. It should be a string. + ''' + return Result(value = {sub_agent["agent_input"]}, agent = {sub_agent_func.replace('get_', '')}) +""" + transfer_funcs_str.append(f"transfer_to_{sub_agent_func.replace('get_', '')}") + transfer_back_to_orchestrator_func += f""" + def transfer_back_to_{agent_name.lower().replace(' ', '_')}({sub_agent["agent_output"]}: str): + ''' + Use this tool to transfer the response back to the `{agent_name}` agent. You can only use this tool when you have tried your best to do the task the orchestrator agent assigned to you. + + Args: + {sub_agent["agent_output"]}: the response to be transferred back to the `{agent_name}` agent. It should be a string. + ''' + return Result(value = {sub_agent["agent_output"]}, agent = {agent_name.lower().replace(' ', '_')}) + {sub_agent_func.replace('get_', '')}.functions.append(transfer_back_to_{agent_name.lower().replace(' ', '_')}) +""" + + agent_func = f"get_{agent_name.lower().replace(' ', '_')}" + + + + create_codes = f"""\ +from autoagent.types import Agent +from autoagent.registry import register_plugin_agent +from autoagent.types import Result + +@register_plugin_agent(name = "{agent_name}", func_name="{agent_func}") +def {agent_func}(model: str): + ''' + {agent_description} + ''' + {import_agent_str} + {instructions_str} + {orchestrator_agent_def} + + {get_sub_agents} + {transfer_sub_agent_func} + {transfer_back_to_orchestrator_func} + + {agent_name.lower().replace(' ', '_')}.functions = [{", ".join(transfer_funcs_str)}] + return {agent_name.lower().replace(' ', '_')} +""" + # print(create_codes) + # with open(f"autoagent/agents/{agent_name.lower().replace(' ', '_')}.py", "w", encoding="utf-8") as f: + # f.write(create_codes) + try: + msg = create_file(agents_dir + "/" + agent_name.lower().replace(' ', '_') + ".py", create_codes, context_variables) + if msg.startswith("Error creating file:"): + return "[ERROR] Failed to create agent. Error: " + msg + result = code_env.run_command('cd {} && python autoagent/agents/{}.py'.format(path, agent_name.lower().replace(' ', '_'))) + if result['status'] != 0: + return "[ERROR] Failed to create agent. Error: " + result['result'] + return "Successfully created agent: " + agent_name + " in " + agents_dir + "/" + agent_name.lower().replace(' ', '_') + ".py" + except Exception as e: + return "[ERROR] Failed to create agent. Error: " + str(e) + +def read_agent(agent_name: str, context_variables: dict): + try: + env: Union[LocalEnv, DockerEnv] = context_variables.get("code_env", LocalEnv()) + try: + path = get_metachain_path(env) + except Exception as e: + return "[ERROR] Failed to get the path of the MetaChain. Error: " + str(e) + agent_list = list_agents(context_variables) + if agent_list.startswith("[ERROR]"): + return "Failed to list agents. Error: " + agent_list + agent_dict = json.loads(agent_list) + if agent_name not in agent_dict.keys(): + return "[ERROR] The agent " + agent_name + " does not exist." + agent_info = agent_dict[agent_name] + ret_val = f"""\ + The information of the agent {agent_name} is: + {agent_info} + """ + return ret_val + except Exception as e: + return "[ERROR] Failed to read the agent. Error: " + str(e) + + +if __name__ == "__main__": +# # print(list_agents({})) +# from litellm import completion +# from autoagent.util import function_to_json +# tools = [function_to_json(create_agent)] +# messages = [ +# {"role": "system", "content": "You are a helpful assistant."}, +# {"role": "user", "content": """\ +# Create an Personalized RAG agent that can answer the question about the given document. There are some tools you can use: +# - save_raw_docs_to_vector_db: Save the raw documents to the vector database. The documents could be: +# - ANY text document with the extension of pdf, docx, txt, etcs. +# - A zip file containing multiple text documents +# - a directory containing multiple text documents +# All documents will be converted to raw text format and saved to the vector database in the chunks of 4096 tokens. +# - query_db: Retrieve information from the database. Use this function when you need to search for information in the database. +# - modify_query: Modify the query based on what you know. Use this function when you need to modify the query to search for more relevant information. +# - answer_query: Answer the user query based on the supporting documents. +# - can_answer: Check if you have enough information to answer the user query. +# - visual_question_answering: This tool is used to answer questions about attached images or videos. + +# There are some global variables you can use: +# glbal_keys | global_vals +# -----------|----------- +# user_name | "Jiabin Tang" +# user_email | "jiabin.tang@gmail.com" + +# [IMPORTANT] NOT ALL tools are required to be used. You can choose the tools that you think are necessary. +# """}, +# ] +# for tool in tools: +# params = tool["function"]["parameters"] +# params["properties"].pop("context_variables", None) +# if "context_variables" in params["required"]: +# params["required"].remove("context_variables") +# # response = completion( +# # model="claude-3-5-sonnet-20241022", +# # messages=messages, +# # tools=tools, +# # tool_choice="auto", # auto is default, but we'll be explicit +# # ) +# # print("\nLLM Response1:\n", response.choices[0].message.tool_calls) +# # args = json.loads(response.choices[0].message.tool_calls[0].function.arguments) +# # create_agent(args["agent_name"], args["agent_description"], args["agent_tools"], args["agent_instructions"], {}) +# # print(list_agents({})) +# print(create_orchestrator_agent("Orchestrator Coding RAG Agent", "An Orchestrator Agent that orchestrates the workflow of the codig agent and the RAG agent.", [{"name": "Personalized RAG Agent", "agent_input": "doc_query", "agent_output": "queried_doc_content"}, {"name": "Coding Agent", "agent_input": "coding_query", "agent_output": "coding_result"}], "You are a helpful assistant.", {})) + docker_cfg = DockerConfig( + container_name = "nl2agent_showcase", + workplace_name = "workplace", + communication_port = 12350, + conda_path = "/root/miniconda3", + local_root = "/Users/tangjiabin/Documents/reasoning/autoagent/workspace_meta_showcase/showcase_nl2agent_showcase" + ) + code_env = DockerEnv(docker_cfg) + context_variables = {"code_env": code_env} + print(run_agent(agent_name='Financial Analysis Orchestrator', query="Based on the 10-K reports of AAPL and MSFT from the past 5 years in the docs directory `docs/aapl-2020-2024-10K/` and `docs/msft-2020-2024-10K/`, along with AAPL's other reports `docs/aapl-other-report/` and available data, conduct a comprehensive horizontal comparison, create a comparative analysis report, and provide constructive investment advice for investing in them in 2025.", ctx_vars='{}', context_variables=context_variables)) + + diff --git a/autoagent/tools/meta/edit_tools.py b/autoagent/tools/meta/edit_tools.py new file mode 100644 index 0000000..51550bc --- /dev/null +++ b/autoagent/tools/meta/edit_tools.py @@ -0,0 +1,222 @@ +from autoagent.registry import registry +from autoagent.environment import LocalEnv, DockerEnv +from typing import Union +from autoagent.tools.terminal_tools import ( + create_file, + create_directory, + run_python, + print_stream, + terminal_page_up, + terminal_page_down, + terminal_page_to, + process_terminal_response + ) +from autoagent.registry import register_tool +import json +def get_metachain_path(env: Union[LocalEnv, DockerEnv]) -> str: + result = env.run_command('pip show autoagent') + if result['status'] != 0: + raise Exception("Failed to list tools. Error: " + result['result']) + stdout = result['result'] + for line in stdout.split('\n'): + if line.startswith('Editable project location:'): + path = line.split(':', 1)[1].strip() + return path + raise Exception("Failed to list tools. The MetaChain is not installed in editable mode.") + +def protect_tools(tool_name: str): + if tool_name in registry.tools_info.keys(): + raise Exception(f"The tool `{tool_name}` can NOT be modified. You can DIRECTLY use the `{tool_name}` tool by USING the `run_tool` tool. Or you can create a new tool using this tool by `from autoagent.tools import {tool_name}`.") + + +@register_tool("list_tools") +def list_tools(context_variables): + """ + List all plugin tools in the MetaChain. + Returns: + A list of information of all plugin tools including name, args, docstring, body, return_type, file_path. + """ + env: Union[LocalEnv, DockerEnv] = context_variables.get("code_env", LocalEnv()) + try: + path = get_metachain_path(env) + except Exception as e: + return "Failed to list tools. Error: " + str(e) + python_code = '"from autoagent.registry import registry; import json; print(\\"TOOL_LIST_START\\"); print(json.dumps(registry.display_plugin_tools_info, indent=4)); print(\\"TOOL_LIST_END\\")"' + list_tools_cmd = f"cd {path} && DEFAULT_LOG=False python -c {python_code}" + result = env.run_command(list_tools_cmd) + if result['status'] != 0: + return "Failed to list tools. Error: " + result['result'] + try: + output = result['result'] + start_marker = "TOOL_LIST_START" + end_marker = "TOOL_LIST_END" + start_idx = output.find(start_marker) + len(start_marker) + end_idx = output.find(end_marker) + + if start_idx == -1 or end_idx == -1: + return "Failed to parse tool list: markers not found" + + json_str = output[start_idx:end_idx].strip() + return json_str + except Exception as e: + return f"Failed to process output: {str(e)}" + # return result['result'] +def check_tool_name(tool_name: str): + if tool_name == "visual_question_answering": + raise Exception("The tool `visual_question_answering` is not allowed to be modified. Directly use the `visual_question_answering` tool to handlen ANY visual tasks.") +@register_tool("create_tool") +def create_tool(tool_name: str, tool_code: str, context_variables): + """ + Create a plugin tool. + Args: + tool_name: The name of the tool. + tool_code: The code of creating the tool. (You should strictly follow the format of the template given to you to create the tool.) + Returns: + A string representation of the result of the tool creation. + """ + # try: + # check_tool_name(tool_name) + # except Exception as e: + # return str(e) + env: Union[LocalEnv, DockerEnv] = context_variables.get("code_env", LocalEnv()) + try: + protect_tools(tool_name) + except Exception as e: + return "[ERROR] Failed to protect the tool `" + tool_name + "` with the following error: " + str(e) + try: + path = get_metachain_path(env) + except Exception as e: + return "[ERROR] Failed to list tools. Error: " + str(e) + + tools_dir = path + "/autoagent/tools" + try: + tool_path = tools_dir + "/" + tool_name + ".py" + if "from autoagent.registry import register_plugin_tool" not in tool_code: + tool_code = "from autoagent.registry import register_plugin_tool\n" + tool_code + msg = create_file(tool_path, tool_code, context_variables) + + if msg.startswith("Error creating file:"): + return "[ERROR] Failed to create tool. Error: " + msg + results = env.run_command(f'cd {path} && python {tool_path}') + if results['status'] != 0: + return "[ERROR] Failed to create tool. The python code of the tool is not correct. Error: " + results['result'] + return "[SUCCESS] Successfully created tool: " + tool_name + " in " + tools_dir + "/" + tool_name + ".py" + except Exception as e: + return "[ERROR] Failed to create tool. Error: " + str(e) + +def tool_exists(tool_name: str, context_variables): + try: + list_res = list_tools(context_variables) + tool_dict = json.loads(list_res) + if tool_name not in tool_dict.keys(): + return False, tool_dict + return True, tool_dict + except Exception as e: + return "Before deleting a tool, you should list all tools first. But the following error occurred: " + str(e), None + +@register_tool("delete_tool") +def delete_tool(tool_name: str, context_variables): + """ + Delete a plugin tool. + Args: + tool_name: The name of the tool to be deleted. + Returns: + A string representation of the result of the tool deletion. + """ + # try: + # check_tool_name(tool_name) + # except Exception as e: + # return str(e) + env: Union[LocalEnv, DockerEnv] = context_variables.get("code_env", LocalEnv()) + # try: + # exist_flag, tool_dict = tool_exists(tool_name, context_variables) + # if isinstance(exist_flag, str): + # return "Before deleting a tool, you should list all tools first. But the following error occurred: " + exist_flag + # if not exist_flag: + # return f"The tool `{tool_name}` does not exist." + # except Exception as e: + # return "Before deleting a tool, you should list all tools first. But the following error occurred: " + str(e) + try: + protect_tools(tool_name) + except Exception as e: + return "[ERROR] Failed to delete the tool `" + tool_name + "` with the following error: " + str(e) + list_res = list_tools(context_variables) + tool_dict = json.loads(list_res) + try: + tool_path = tool_dict[tool_name]['file_path'] + except KeyError: + return "The tool `" + tool_name + "` does not exist." + except Exception as e: + return "Error: " + str(e) + try: + result = env.run_command(f"rm {tool_path}") + if result['status'] != 0: + return f"[ERROR] Failed to delete tool: `{tool_name}`. Error: " + result['result'] + return f"[SUCCESS] Successfully deleted tool: `{tool_name}`." + except Exception as e: + return f"[ERROR] Failed to delete tool: `{tool_name}`. Error: " + str(e) + + +@register_tool("run_tool") +@process_terminal_response +def run_tool(tool_name: str, run_code: str, context_variables): + """ + Run a tool with the given code. + + Args: + tool_name: The name of the tool to be run. + run_code: The code to be run. + Returns: + A string representation of the result of the tool running. + """ + env: Union[LocalEnv, DockerEnv] = context_variables.get("code_env", LocalEnv()) + try: + path = get_metachain_path(env) + except Exception as e: + return "Failed to get the path of the MetaChain. Error: " + str(e) + # try: + # exist_flag, tool_dict = tool_exists(tool_name, context_variables) + # if isinstance(exist_flag, str): + # return "Before deleting a tool, you should list all tools first. But the following error occurred: " + exist_flag + # if not exist_flag: + # return f"The tool `{tool_name}` does not exist." + # except Exception as e: + # return "Before testing a tool, you should list all tools first. But the following error occurred: " + str(e) + + test_dir = path + "/test_tools" + + try: + msg = create_directory(test_dir, context_variables) + if msg.startswith("Error creating directory:"): + return "[ERROR] Failed to create the test directory. Error: " + msg + except Exception as e: + return "[ERROR] Failed to create the test directory. Error: " + str(e) + + test_file_path = test_dir + "/" + "test_" + tool_name + ".py" + try: + msg = create_file(test_file_path, run_code, context_variables) + if msg.startswith("Error creating file:"): + return "[ERROR] Failed to create the test file. Error: " + msg + except Exception as e: + return "[ERROR] Failed to create the test file. Error: " + str(e) + + try: + # result = run_python(context_variables, test_file_path, cwd=path, env_vars={"DEFAULT_LOG": "False"}) + # if "[SUCCESS]" not in result: + # return "[ERROR] Failed to test the tool. The test case is not correct. The result is: " + result + # return f"The result is of the tool `{tool_name}`: \n{result.replace('[SUCCESS]', '')}" + run_cmd = f"cd {path} && DEFAULT_LOG=False python {test_file_path}" + result = env.run_command(run_cmd, print_stream) + return result + except Exception as e: + return "[ERROR] Failed to test the tool. Error: " + str(e) + +if __name__ == "__main__": + # print(list_tools({})) + # print(create_tool("visual_question_answering", "print('Hello, World!')", {})) + test_code = """ +from autoagent.tools import test_file_tools +print(test_file_tools()) +""" + print(run_tool("test_file_tools", test_code, {})) + print(terminal_page_down()) diff --git a/autoagent/tools/meta/edit_workflow.py b/autoagent/tools/meta/edit_workflow.py new file mode 100644 index 0000000..99b8bd9 --- /dev/null +++ b/autoagent/tools/meta/edit_workflow.py @@ -0,0 +1,301 @@ +from typing import Union +from autoagent.environment import LocalEnv, DockerEnv +from autoagent.tools.meta.edit_tools import get_metachain_path +from autoagent.tools.meta.edit_agents import list_agents +from autoagent.tools.terminal_tools import create_file, create_directory, print_stream, process_terminal_response +from autoagent.registry import register_tool +import json +from autoagent import MetaChain +from autoagent.types import Response +import shlex +from datetime import datetime +from pydantic import BaseModel +CODE_PREFIX = """\ +import asyncio +import json +import argparse +from openai import AsyncOpenAI +from openai.types.chat import ChatCompletionMessageToolCall +from autoagent.flow import default_drive, EventInput, ReturnBehavior +from autoagent.flow.dynamic import goto_events, abort_this +import re +from autoagent import MetaChain +from autoagent.types import Response +from autoagent.registry import register_workflow + +def extract_answer(response: str, key: str): + pattern = f"<{key}>(.*?)" + matches = re.findall(pattern, response, re.DOTALL) + return matches[0] if len(matches) > 0 else None +""" + + +CODE_MAIN = """ +@register_workflow(name = '{workflow_name}') +async def {workflow_name}(system_input: str): + storage_results = dict({input_key} = system_input) + await default_drive.invoke_event( + on_start, + global_ctx=storage_results, + ) + system_output = storage_results.get({output_key}, None) + return system_output +""" + +EVENT_TEMPLATE_PREFIX = """\ +@default_drive.{event_method} +async def {event_name}(event: EventInput, global_ctx): + inputs = {inputs} + input_dict = dict() + for inp in inputs: + input_dict[inp["key"]] = global_ctx.get(inp["key"], None) + + messages = global_ctx.get('messages', []) + task = {task} + outputs = {output_list} + agent = {agent_func_name}({model}) + +""" +EVENT_TEMPLATE_FIX = r""" + input_str = [] + for key, value in input_dict.items(): + input_str.append(f"The {key.replace('_', ' ')} is {value}") + input_str = "\n".join(input_str) + "\n" + query = input_str + '.\nThe task is: ' + task + '.\n' +""" + +# QUERY_TEMPLATE = """\ +# query = input_str + '.\\nThe task is: ' + task + '.\\n' +# """ + +START_EVENT_CODE = """\ +@default_drive.make_event +async def on_start(event: EventInput, global_ctx): + print("start the workflow:" + {workflow_name}) +""" + + + + +IF_ELSE_SUFFIX = \ +""" +You should follow the above instructions, and return the result in the following format: +""" + +EVENT_TEMPLATE_SUFFIX = """\ + messages.append({ + "role": "user", + "content": query + }) + client = MetaChain() + response: Response = await client.run_async(agent = agent, messages = messages, context_variables = global_ctx, debug = True) + result = response.messages[-1]["content"] + messages.extend(response.messages) + global_ctx["messages"] = messages + + for output in outputs: + ans = extract_answer(result, output["key"]) + if ans: + if output["action"]["type"] == "RESULT": + global_ctx[output["key"]] = ans + return ans + elif output["action"]["type"] == "ABORT": + return abort_this() + elif output["action"]["type"] == "GO_TO": + return goto_events([output["action"]["value"]]) + elif len(outputs) == 1: + global_ctx[output["key"]] = result + return result + raise Exception("No valid answer found") +""" + + +def start_event_to_code(workflow_name: str) -> str: + """ + Convert the start event to code. + """ + return START_EVENT_CODE.format(workflow_name = repr(workflow_name)) + +def single_event_to_code(event: dict, agent_info_dict: dict) -> str: + """ + Convert a single event to code. + + A event contains: + - name (str): the name of the event + - input (dict): the input to the event + - task (str): the task to perform + - outputs (list[dict]): the outputs to the event + - listen (list[str]): the listen to the event + - agent (dict): the agent to run + """ + if event["listen"] == None or len(event["listen"]) == 0: + event_method = "make_event" + else: + event_method = "listen_group([{}])".format(", ".join(event["listen"])) + inputs = event["inputs"] + + event_code = EVENT_TEMPLATE_PREFIX.format(event_method = event_method, event_name = event["name"], inputs = inputs, task = repr(event["task"]), output_list = event["outputs"], agent_mode_name = agent_info_dict[event["agent"]["name"]]["mode_name"], agent_func_name = agent_info_dict[event["agent"]["name"]]["func_name"], model = repr(event["agent"]["model"])) + EVENT_TEMPLATE_FIX + + if len(event["outputs"]) > 1: + condition_str = [] + for output in event["outputs"]: + condition_str.append(f"If {output['condition']}, then encapsulate your final answer (answer ONLY) within <{output['key']}> and . ") + query_suffix = "\n".join(condition_str) + query_suffix = f""" + query_suffix = {repr(IF_ELSE_SUFFIX)} + query_suffix += {repr(query_suffix)} + query += query_suffix +""" + event_code += query_suffix + EVENT_TEMPLATE_SUFFIX + else: + event_code += EVENT_TEMPLATE_SUFFIX + + return event_code + +@register_tool("create_workflow") +def create_workflow(workflow_name: str, context_variables: dict) -> str: + workflow_form = context_variables.get("workflow_form", None) + if workflow_form is None: + return "Failed to get workflow form. Please provide a workflow form." + workflow_form = workflow_form.model_dump() if isinstance(workflow_form, BaseModel) else workflow_form + assert workflow_name == workflow_form['name'], "The workflow name must be the same as the name in the workflow form." + system_input = workflow_form['system_input'] + system_output = workflow_form['system_output'] + code_env: Union[LocalEnv, DockerEnv] = context_variables.get("code_env", LocalEnv()) + try: + path = get_metachain_path(code_env) + except Exception as e: + return "[ERROR] Failed to list agents. Error: " + str(e) + + workflows_dir = path + "/autoagent/workflows" + agent_list = list_agents(context_variables) + if agent_list.startswith("[ERROR]"): + return "Failed to list agents. Error: " + agent_list + agent_dict = json.loads(agent_list) + agent_info_dict = {} + workflow_name = workflow_form["name"] + for a in workflow_form["agents"]: + agent_info_dict[a["name"]] = {"name": a["name"], "func_name": agent_dict[a["name"]]["func_name"], "mode_name": a["name"].replace(" ", "_").lower()} + + import_agent_str = "" + for ainfo in agent_info_dict.values(): + import_agent_str += f""" +from autoagent.agents import {ainfo['func_name']} +""" + events = workflow_form["events"] + events_code = CODE_PREFIX + import_agent_str + for event in events: + if event["name"] == "on_start": + events_code += start_event_to_code(workflow_name) + else: + events_code += single_event_to_code(event, agent_info_dict) + + + events_code += CODE_MAIN.format(workflow_name = workflow_name, input_key = system_input["key"], output_key = repr(system_output["key"])) + + try: + msg = create_file(workflows_dir + "/" + workflow_name.lower().replace(' ', '_') + "_flow.py", events_code, context_variables) + if msg.startswith("Error creating file:"): + return "[ERROR] Failed to create workflow. Error: " + msg + result = code_env.run_command('cd {} && python autoagent/workflows/{}_flow.py'.format(path, workflow_name.lower().replace(' ', '_'))) + if result['status'] != 0: + return "[ERROR] Failed to create workflow. Error: " + result['result'] + return "Successfully created workflow: " + workflow_name + " in " + workflows_dir + "/" + workflow_name.lower().replace(' ', '_') + "_flow.py" + except Exception as e: + return "[ERROR] Failed to create workflow. Error: " + str(e) + +@register_tool("list_workflows") +def list_workflows(context_variables): + """ + List all workflows in the MetaChain. + Returns: + A list of information of all workflows including name, args, docstring, body, return_type, file_path. + """ + env: Union[LocalEnv, DockerEnv] = context_variables.get("code_env", LocalEnv()) + try: + path = get_metachain_path(env) + except Exception as e: + return "[ERROR] Failed to list workflows. Error: " + str(e) + python_code = '"from autoagent.registry import registry; import json; print(\\"WORKFLOW_LIST_START\\"); print(json.dumps(registry.display_workflows_info, indent=4)); print(\\"WORKFLOW_LIST_END\\")"' + list_workflows_cmd = f"cd {path} && DEFAULT_LOG=False python -c {python_code}" + result = env.run_command(list_workflows_cmd) + if result['status'] != 0: + return "[ERROR] Failed to list workflows. Error: " + result['result'] + try: + output = result['result'] + start_marker = "WORKFLOW_LIST_START" + end_marker = "WORKFLOW_LIST_END" + start_idx = output.find(start_marker) + len(start_marker) + end_idx = output.find(end_marker) + + if start_idx == -1 or end_idx == -1: + return "[ERROR] Failed to parse workflow list: markers not found" + + json_str = output[start_idx:end_idx].strip() + return json_str + except Exception as e: + return f"[ERROR] Failed to process output: {str(e)}" + +@register_tool("run_workflow") +@process_terminal_response +def run_workflow(workflow_name: str, system_input: str, context_variables: dict) -> str: + env: Union[LocalEnv, DockerEnv] = context_variables.get("code_env", LocalEnv()) + try: + path = get_metachain_path(env) + except Exception as e: + return "[ERROR] Failed to get the path of the MetaChain. Error: " + str(e) + try: + workflow_list = list_workflows(context_variables) + if workflow_list.startswith("[ERROR]"): + return "[ERROR] Failed to list workflows. Error: " + workflow_list + workflow_dict = json.loads(workflow_list) + if workflow_name in workflow_dict.keys(): + workflow_info = workflow_dict[workflow_name] + workflow_func = workflow_info['func_name'] + else: + return "[ERROR] The workflow " + workflow_name + " does not exist." + except Exception as e: + return "[ERROR] Before running a agent, you should list all agents first. But the following error occurred: " + str(e) + + try: + # query = shlex.quote(query) + # run_cmd = f'cd {path} && DEFAULT_LOG=False mc agent --model={model} --agent_func={agent_func} --query={query} {ctx_vars_str}' + system_input = shlex.quote(system_input) + # timestamp = datetime.now().strftime("%Y%m%d_%H%M%S") + # create_directory(f"{path}/tmp_input", context_variables) + # input_file = f"{path}/tmp_input/input_{timestamp}.txt" + # create_file(input_file, system_input, context_variables) + shell_content = f"""#!/bin/bash +cd {path} +DEFAULT_LOG=False mc workflow --workflow_name={workflow_name} --system_input={system_input} +""" + timestamp = datetime.now().strftime("%Y%m%d_%H%M%S") + create_directory(f"{path}/tmp_shell", context_variables) + create_file(f"{path}/tmp_shell/run_workflow_{timestamp}.sh", shell_content, context_variables) + run_cmd = f"cd {path} && chmod +x tmp_shell/run_workflow_{timestamp}.sh && ./tmp_shell/run_workflow_{timestamp}.sh" + result = env.run_command(run_cmd, print_stream) + return result + except Exception as e: + return "[ERROR] Failed to run the workflow. Error: " + str(e) + +if __name__ == "__main__": + from autoagent.environment import DockerConfig, DockerEnv, LocalEnv + docker_cfg = DockerConfig( + container_name = "nl2agent_showcase", + workplace_name = "workplace", + communication_port = 12350, + conda_path = "/root/miniconda3", + local_root = "/Users/tangjiabin/Documents/reasoning/autoagent/workspace_meta_showcase/showcase_nl2agent_showcase" + ) + code_env = DockerEnv(docker_cfg) + with open("/Users/tangjiabin/Documents/reasoning/autoagent/autoagent/agents/meta_agent/workflow_form/condition_mining.json", 'r', encoding='utf-8') as f: + workflow_form = json.load(f) + print(workflow_form) + + context_variables = {"workflow_form": workflow_form, "code_env": code_env} + result = create_workflow(workflow_form["name"], context_variables) + print(result) + + result = run_workflow(workflow_form["name"], 'The wheel shown is spun twice, so that the numbers indicated by the pointer are randomly determined (with each number on the wheel being equally likely). The two numbers determined in this way are recorded. The first number is divided by 4, determining one of the remainders 1,2,3 marking the columns of the checkerboard shown. The second number is divided by 5, determining one of the remainders 1,2,3,4 marking the rows of the checkerboard. Finally, a checker is placed on the square where this column and row meet. What is the probability that the checker is placed on a shaded square of the checkerboard? [asy] unitsize(1cm); draw(Circle((0,0),2),linewidth(0.7)); draw((1.7,1)--(-1.7,-1),linewidth(0.7)); draw((1.7,-1)--(-1.7,1),linewidth(0.7)); draw((0,2)--(0,-2)); label("1",(0.8,0.5),NW); label("2",(0.8,-0.5),SW); label("6",(-0.8,0.5),NE); label("9",(-0.8,-0.5),SE); label("3",(-0.7,0),W); label("7",(0.7,0),E); draw((-2.8,0)--(-2.1,0),Arrow); label("Pointer",(-2.8,0),W); fill((3,0)--(3,1)--(4,1)--(4,0)--cycle,gray(0.7)); fill((3,-2)--(3,-1)--(4,-1)--(4,-2)--cycle,gray(0.7)); fill((4,1)--(4,2)--(5,2)--(5,1)--cycle,gray(0.7)); fill((4,-1)--(4,0)--(5,0)--(5,-1)--cycle,gray(0.7)); fill((5,0)--(5,1)--(6,1)--(6,0)--cycle,gray(0.7)); fill((5,-2)--(5,-1)--(6,-1)--(6,-2)--cycle,gray(0.7)); draw((3,-2)--(3,2)--(6,2)--(6,-2)--cycle,linewidth(0.7)); draw((3,-1)--(6,-1),linewidth(0.7)); draw((3,0)--(6,0),linewidth(0.7)); draw((3,1)--(6,1),linewidth(0.7)); draw((4,-2)--(4,2),linewidth(0.7)); draw((5,-2)--(5,2),linewidth(0.7)); label("1",(3.5,-2),S); label("2",(4.5,-2),S); label("3",(5.5,-2),S); label("1",(3,-1.5),W); label("2",(3,-0.5),W); label("3",(3,0.5),W); label("4",(3,1.5),W); [/asy]', context_variables) + print(result) + \ No newline at end of file diff --git a/autoagent/tools/meta/search_tools.py b/autoagent/tools/meta/search_tools.py new file mode 100644 index 0000000..40d98a2 --- /dev/null +++ b/autoagent/tools/meta/search_tools.py @@ -0,0 +1,106 @@ +from autoagent.registry import register_tool +from huggingface_hub import HfApi, hf_hub_download +from typing import List +import tempfile +import os + +@register_tool("search_trending_models_on_huggingface") +def search_trending_models_on_huggingface(pipeline_tag: str, limit: int = 5) -> str: + """ + Search trending models on Hugging Face. Use this tool when you want to create a tool that uses Hugging Face models, only support the following tags: ['audio-text-to-text', 'text-to-image', 'image-to-image', 'image-to-video', 'text-to-video', 'text-to-speech', 'text-to-audio', 'automatic-speech-recognition', 'audio-to-audio']. + + Args: + pipeline_tag: The pipeline tag you want to search on Hugging Face. ONLY support the following tags: ['audio-text-to-text', 'text-to-image', 'image-to-image', 'image-to-video', 'text-to-video', 'text-to-speech', 'text-to-audio', 'automatic-speech-recognition', 'audio-to-audio']. + limit: The number of models you want to search on Hugging Face. + Returns: + A string representation of the information you found on Hugging Face. + """ + # if pipeline_tag in ['image-text-to-text', 'visual-question-answering', 'video-text-to-text']: + # return f"As for the tags {pipeline_tag}, you should use `visual_question_answering` tool instead!" + if pipeline_tag not in ['audio-text-to-text', 'text-to-image', 'image-to-image', 'image-to-video', 'text-to-video', 'text-to-speech', 'text-to-audio', 'automatic-speech-recognition', 'audio-to-audio']: + return f"Only the following tags are supported: ['audio-text-to-text', 'text-to-image', 'image-to-image', 'image-to-video', 'text-to-video', 'text-to-speech', 'text-to-audio', 'automatic-speech-recognition', 'audio-to-audio']. If you want to use ['image-text-to-text', 'visual-question-answering', 'video-text-to-text'], you should use `visual_question_answering` tool instead!" + api = HfApi() + + # 搜索模型和数据集 + models = api.list_models(pipeline_tag=pipeline_tag, limit=limit) + + # 格式化结果 + result = [] + + # 添加模型信息 + result.append("Finding models on Hugging Face:") + for model in models: + result.append(f"- Model ID: {model.id}") + + # 收集模型信息 + info = [] + if model.card_data: + if model.card_data.language: + info.append(f"Language: {model.card_data.language}") + if model.card_data.license: + info.append(f"License: {model.card_data.license}") + if model.card_data.library_name: + info.append(f"Framework: {model.card_data.library_name}") + if model.card_data.pipeline_tag: + info.append(f"Task: {model.card_data.pipeline_tag}") + + if model.tags: + info.append(f"Tags: {', '.join(model.tags)}") + if model.downloads: + info.append(f"Downloads(30 days): {model.downloads}") + + # 添加收集到的信息 + if info: + result.append(" " + "\n ".join(info)) + + # 尝试获取README内容 + try: + with tempfile.TemporaryDirectory() as tmp_dir: + readme_path = hf_hub_download( + repo_id=model.id, + filename="README.md", + repo_type="model", + local_dir=tmp_dir + ) + with open(readme_path, 'r', encoding='utf-8') as f: + readme_content = f.read() + # 提取前500个字符作为简介 + summary = readme_content[:500].strip() + "..." + result.append(" Summary: " + summary.replace('\n', ' ')) + except Exception as e: + result.append(" Summary: Failed to get") + + result.append("") + + return "\n".join(result) + +@register_tool("get_hf_model_tools_doc") +def get_hf_model_tools_doc(model_id: str) -> str: + """ + Get the detailed information of a model on Hugging Face, such as the detailed usage of the model containing the model's README.md. You should use this tool after you have used `search_trending_models_on_huggingface` to find the model you want to use. + + Args: + model_id: The model id you want to get the detailed information on Hugging Face. + Returns: + A string representation of the detailed information of the model. + """ + result = [] + try: + with tempfile.TemporaryDirectory() as tmp_dir: + readme_path = hf_hub_download( + repo_id=model_id, + filename="README.md", + repo_type="model", + local_dir=tmp_dir + ) + with open(readme_path, 'r', encoding='utf-8') as f: + readme_content = f.read() + summary = readme_content.strip() + result.append("The detailed usage of the model is: " + summary) + except Exception as e: + result.append("Failed to get the detailed usage of the model. Error: " + str(e)) + return "\n".join(result) + +if __name__ == "__main__": + print(search_trending_models_on_huggingface("automatic-speech-recognition", limit=5)) + diff --git a/autoagent/tools/meta/tool_retriever.py b/autoagent/tools/meta/tool_retriever.py new file mode 100644 index 0000000..9c49b67 --- /dev/null +++ b/autoagent/tools/meta/tool_retriever.py @@ -0,0 +1,38 @@ +from autoagent.memory.tool_memory import ToolMemory, ToolReranker +import os +from autoagent.io_utils import get_file_md5 +import pandas as pd +from autoagent.registry import register_tool +from constant import COMPLETION_MODEL, EMBEDDING_MODEL + +@register_tool("get_api_plugin_tools_doc") +def get_api_plugin_tools_doc(query_text: str) -> str: + """ + Retrieve satisfied tool documents based on the query text. + Args: + query_text: A query or request from users and you need to find the satisfied tool documents based on the query text. + Returns: + A string representation of the reranked results. + """ + platform = 'default' + tool_memory = ToolMemory(project_path = './code_db', db_name = ".tool_table_" + platform, platform=platform, api_key=os.getenv("OPENAI_API_KEY"), embedding_model=EMBEDDING_MODEL) + # tool_reranker = ToolReranker(model="gpt-4o-2024-08-06") + tool_reranker = ToolReranker(model=COMPLETION_MODEL) + tool_path = "./tool_docs.csv" + code_id = get_file_md5(tool_path) + # print(code_id) + tool_memory.collection_name = tool_memory.collection_name + f"_{code_id}" + if tool_memory.count() == 0: + tool_memory.add_dataframe(pd.read_csv(tool_path), batch_size=100) + res_df = tool_memory.query_table(query_text, n_results=5) + # print(res_df) + try: + reranked_result = tool_reranker.dummy_rerank(query_text, res_df) + except Exception as e: + return "Failed to rerank the tool documentation. Error: " + str(e) + + return reranked_result + +if __name__ == "__main__": + os.environ["GEMINI_API_KEY"] = "AIzaSyDblGdaCwhWq0RpXe7aCPFQr0MBg__GN2E" + print(get_api_plugin_tools_doc("Youtube")) \ No newline at end of file diff --git a/autoagent/tools/rag_code.py b/autoagent/tools/rag_code.py new file mode 100644 index 0000000..1408033 --- /dev/null +++ b/autoagent/tools/rag_code.py @@ -0,0 +1,32 @@ +from autoagent.memory.code_memory import CodeMemory, CodeReranker +import os +from autoagent.environment import DockerEnv, LocalEnv +from autoagent.io_utils import compress_folder, get_file_md5 +from autoagent.registry import register_tool +from typing import Union +@register_tool("code_rag") +def code_rag(query_text: str, context_variables) -> str: + """ + Retrieve information from a code directory. Use this function when there is a need to search for information in the codebase. + Args: + query_text: Anything you want to search in the code directory, like a function name, a class name, a variable name, etc. + Returns: + A string representation of the reranked results. + """ + env: Union[DockerEnv, LocalEnv] = context_variables.get("code_env", LocalEnv()) + code_memory = CodeMemory(project_path = './code_db', platform='OpenAI', api_key=os.getenv("OPENAI_API_KEY"),embedding_model='text-embedding-3-small') + code_reranker = CodeReranker(model="gpt-4o-2024-08-06") + code_path = f"{env.local_workplace}/autoagent" + compress_folder(code_path, f"{env.local_workplace}/", "autoagent.zip") + code_id = get_file_md5(f"{env.local_workplace}/autoagent.zip") + code_memory.collection_name = code_memory.collection_name + f"_{code_id}" + + if code_memory.count() == 0: + code_memory.add_code_files(f"{env.local_workplace}/autoagent", exclude_prefix=['__pycache__', 'code_db', '.git']) + + query_results = code_memory.query_code(query_text, n_results=20) + reranked_results = code_reranker.rerank(query_text, query_results) + return reranked_results + + + diff --git a/autoagent/tools/rag_tools.py b/autoagent/tools/rag_tools.py new file mode 100644 index 0000000..e819648 --- /dev/null +++ b/autoagent/tools/rag_tools.py @@ -0,0 +1,276 @@ +from autoagent.memory.rag_memory import Memory +import os +from autoagent.environment.docker_env import DockerEnv +from autoagent.environment.local_env import LocalEnv +from typing import Union +from autoagent.environment.local_env import LocalEnv +from autoagent.io_utils import compress_folder, get_file_md5 +from autoagent.registry import register_tool, register_plugin_tool +from litellm import completion +import zipfile +import shutil +from autoagent.environment.markdown_browser.mdconvert import MarkdownConverter +from autoagent.memory.utils import chunking_by_token_size +import math +from autoagent.types import Result +# @register_tool("load_db") +# def load_db(db_path: str) -> str: + +@register_tool("save_raw_docs_to_vector_db") +@register_plugin_tool("save_raw_docs_to_vector_db") +def save_raw_docs_to_vector_db(context_variables: dict, doc_name: str, saved_vector_db_name: str, overwrite: bool = False) -> Result: + """ + Save the raw documents to the vector database. The documents could be: + - ANY text document with the extension of pdf, docx, txt, etcs. + - A zip file containing multiple text documents + - a directory containing multiple text documents + All documents will be converted to raw text format and saved to the vector database in the chunks of 4096 tokens. + + Args: + doc_name: The name of the raw documents. All documents will be stored in the the directory: /workplace/docs. + [NOTES] doc_name should be the name of the file or directory, not the path to the file or directory, which means `docs/dir_name/` is not a valid doc_name. + saved_vector_db_name: the name of collection you want to save the documents to. + overwrite: Whether to overwrite the existing vector database when the vector database of the documents already exists. (default: False) + """ + try: + memo: Memory = context_variables.get("memo", Memory(project_path=os.path.join(os.getcwd(), "user_db"), db_name = ".user_db")) + assert memo is not None, "memo is not set" + code_env: Union[DockerEnv, LocalEnv] = context_variables.get("code_env", LocalEnv()) + assert code_env is not None, "code_env is not set" + + # check if the saved_vector_db_name is already in the vector database + if memo.count(saved_vector_db_name) > 0: + if overwrite: + prefix_res = f"[WARNING] The collection `{saved_vector_db_name}` of the vector database already exists. Overwriting the existing collection." + else: + return f"[WARNING] The collection `{saved_vector_db_name}` of the vector database already exists. Please set the overwrite flag to True if you want to overwrite the existing collection." + else: + prefix_res = "" + + doc_dir = os.path.join(code_env.local_workplace, "docs") + os.makedirs(doc_dir, exist_ok=True) + if doc_name.startswith("docs/"): + doc_name = doc_name.replace("docs/", "") + elif doc_name.startswith("/workspace/docs/"): + doc_name = doc_name.replace("/workspace/docs/", "") + doc_path = os.path.join(doc_dir, doc_name) + assert os.path.exists(doc_path), f"The document `{doc_name}` does not exist in the directory `/workplace/docs`" + # the doc_path is a directory + if os.path.isdir(doc_path): + file_list = [] + for file in os.listdir(doc_path): + if file.endswith(('.pdf', '.docx', '.txt')): + file_list.append(os.path.join(doc_path, file)) + # the doc_path is a zip file + elif os.path.isfile(doc_path) and (doc_path.endswith('.zip') or doc_path.endswith('.tar') or doc_path.endswith('.tar.gz')): + file_name = os.path.splitext(doc_name)[0] + extract_dir = os.path.join(doc_dir, file_name) + os.makedirs(extract_dir, exist_ok=True) + + with zipfile.ZipFile(doc_path, 'r') as zip_ref: + zip_ref.extractall(extract_dir) + + # 将提取的文件路径添加到file_list中 + file_list = [] + for root, _, files in os.walk(extract_dir): + for file in files: + if file.endswith(('.pdf', '.docx', '.txt')): + file_list.append(os.path.join(root, file)) + # the doc_path is a single file + elif os.path.isfile(doc_path): + file_list.append(doc_path) + else: + raise ValueError(f"The document `{doc_name}` is not a valid file or directory") + + mdconvert = MarkdownConverter() + + ret_val = prefix_res + batch_size = 200 + for file in file_list: + queries = [] + doc_content = mdconvert.convert_local(file).text_content + content_chunks = chunking_by_token_size(doc_content, max_token_size=4096) + idx_list = ["chunk_" + str(chunk['chunk_order_index']) for chunk in content_chunks] + for chunk in content_chunks: + query = { + 'query': chunk['content'], + 'response': f"The {chunk['chunk_order_index']} chunk of the content of the file {file} is: \n{chunk['content']}" + } + queries.append(query) + + num_batches = math.ceil(len(queries) / batch_size) + + for i in range(num_batches): + start_idx = i * batch_size + end_idx = min((i + 1) * batch_size, len(queries)) + batch_queries = queries[start_idx:end_idx] + batch_idx = idx_list[start_idx:end_idx] + + # Add the current batch of queries + memo.add_query(batch_queries, collection=saved_vector_db_name, idx=batch_idx) + ret_val += f"The {file} has been added to the vector database `{saved_vector_db_name}`." + context_variables["memo"] = memo + return Result( + value=ret_val, + context_variables=context_variables + ) + except Exception as e: + ret_val = f"[ERROR] Failed to save the raw documents to the vector database: {e}" + return ret_val + +@register_tool("query_db") +@register_plugin_tool("query_db") +def query_db(context_variables: dict, query_text: str, saved_vector_db_name: str, n_results: int = 5) -> str: + """ + Retrieve information from the database. Use this function when you need to search for information in the database. + + Args: + query_text: The query to search for information in the database. + saved_vector_db_name: The name of the vector database to search for information. + n_results: The number of results to return. (default: 5) + Returns: + A string representation of the queried results. + """ + try: + memo: Memory = context_variables.get("memo", Memory(project_path=os.path.join(os.getcwd(), "user_db"), db_name = ".user_db")) + assert memo is not None, "memo is not set" + if memo.count(saved_vector_db_name) == 0: + return f"[ERROR] The vector database `{saved_vector_db_name}` does not exist. Please use function `save_raw_docs_to_vector_db` to save the raw documents to the vector database." + results = memo.query([query_text], collection=saved_vector_db_name, n_results=n_results) + + metadata_results = results['metadatas'][0] + results = [item['response'] for item in metadata_results] + ret_val = "\n".join(results) + except Exception as e: + ret_val = f"[ERROR] Failed to query the vector database: {e}" + finally: + return ret_val + +@register_tool("modify_query") +@register_plugin_tool("modify_query") +def modify_query(what_you_know: str, query_text: str, **kwargs) -> str: + """ + Modify the query based on what you know. Use this function when you need to modify the query to search for more relevant information. + + Args: + what_you_know: The knowledge you have about the case. + query_text: The original query. + Returns: + The modified query. + """ + system_prompt = \ + f""" + + Assume you are an assistant searching for information. Now that you already know some knowledge ([What you know]), what sub-questions ([Modified query]) do you need to search for to help you answer the question ([Query]) you want to explore. + + Modify the query based on what you know, here is some example: + Example 1: + [What you know]: Alice and Bob have lunch together at 12:00 PM. + [Query]: What did Alice and Bob do after the lunch? + [Modified query]: What did Alice and Bob do after 12:00 PM? + + Example 2: + [What you know]: Alice and Bob went to the cinema yesterday. + [Query]: What did Alice and Bob do after the cinema? + [Modified query]: What did Alice and Bob do yesterday? + + Return only 1 modified query. + """ + + user_prompt = f""" + What you know: {what_you_know} + Query: {query_text} + Modified query: + """ + create_params = { + "model": "gpt-4o-mini", + "messages": [ + {"role": "system", "content": system_prompt}, + {"role": "user", "content": user_prompt} + ], + "stream": False, + } + response = completion(**create_params) + modified_query = response.choices[0].message.content + return f"The modified query is: {modified_query}. Now use function `query_db` to search the related information in the DataBase." + + + +@register_tool("answer_query") +@register_plugin_tool("answer_query") +def answer_query(original_user_query: str, supporting_docs: str, **kwargs) -> str: + """ + Answer the user query based on the supporting documents. + + Args: + original_user_query: The original user query. + supporting_docs: The supporting documents. + Returns: + The answer to the user query. + """ + system_prompt = \ + f""" + You are a helpful assistant. Answer the user query based on the supporting documents. + If you have not found the answer, say "Insufficient information." + """ + + user_prompt = f""" + + Here is the original user query and the supporting documents: + Original user query: {original_user_query} + Supporting documents: {supporting_docs} + Answer: + """ + create_params = { + "model": "gpt-4o-mini", + "messages": [ + {"role": "system", "content": system_prompt}, + {"role": "user", "content": user_prompt} + ], + "stream": False, + } + response = completion(**create_params) + answer = response.choices[0].message.content + + + return answer + + +@register_tool("can_answer") +@register_plugin_tool("can_answer") +def can_answer(user_query: str, supporting_docs: str, **kwargs) -> str: + """ + Check if you have enough information to answer the user query. + + Args: + user_query: The user query. + supporting_docs: The supporting documents. + Returns: + "True" if you have enough information to answer the user query, "False" otherwise. + """ + system_prompt = \ + f""" + You are a helpful assistant. Check if you have enough information to answer the user query. + The answer should only be "True" or "False". + """ + + user_prompt = f""" + + Here is the original user query and the supporting documents: + Original user query: {user_query} + Supporting documents: {supporting_docs} + Answer: + """ + create_params = { + "model": "gpt-4o-mini", + "messages": [ + {"role": "system", "content": system_prompt}, + {"role": "user", "content": user_prompt} + ], + "stream": False, + } + response = completion(**create_params) + answer = response.choices[0].message.content + + return answer + diff --git a/autoagent/tools/terminal_tools.py b/autoagent/tools/terminal_tools.py new file mode 100644 index 0000000..fcc8fb1 --- /dev/null +++ b/autoagent/tools/terminal_tools.py @@ -0,0 +1,441 @@ + +import os +import socket +import json +import base64 +import math +# from autoagent.util import run_command_in_container +from autoagent.environment.docker_env import DockerEnv, DockerConfig +from autoagent.registry import register_tool +from autoagent.environment.markdown_browser.requests_markdown_browser import RequestsMarkdownBrowser +from typing import Tuple, Optional, Dict, Union +import time +import tiktoken +from datetime import datetime +from functools import wraps +from rich.console import Console +from pathlib import Path +from autoagent.environment.local_env import LocalEnv +from autoagent.environment.docker_env import DockerEnv +import inspect + +terminal_env = RequestsMarkdownBrowser(local_root=os.getcwd(), workplace_name="terminal_env", viewport_size=1024 * 8) + +def _get_browser_state(env: RequestsMarkdownBrowser) -> Tuple[str, str]: + """ + Get the current state of the browser, including the header and content. + """ + # print(env.address) + address = env.address + tool_name = address.split('/')[-1].split('.')[0].split('___')[-1] + header = f"[The output of the tool `{tool_name}` showing in the interactive terminal]\n" + + current_page = env.viewport_current_page + total_pages = len(env.viewport_pages) + + + for i in range(len(env.history) - 2, -1, -1): # Start from the second last + if env.history[i][0] == address: + header += f"You previously visited this page of terminal {round(time.time() - env.history[i][1])} seconds ago.\n" + break + prefix = f"[Your terminal is currently open to the page '{env.page_title}']\n" if env.page_title is not None else "" + + header = prefix + header + header += f"Terminal viewport position: Showing page {current_page+1} of {total_pages}.\n" + if total_pages > 1: + header += f"[NOTE] The output of the tool `{tool_name}`, you can use `terminal_page_up` to scroll up and `terminal_page_down` to scroll down. If there are many pages with meaningless content like progress bar or output of generating directory structure when there are many datasets in the directory, you can use `terminal_page_to` to move the viewport to the end of terminal where the meaningful content is.\n" + return (header, env.viewport) + +def open_local_terminal_output(path: str): + """ + Open a local file at a path in the text-based browser and return current viewport content. + + Args: + path: The absolute path of a local file to visit. + """ + try: + # assert DOCKER_WORKPLACE_NAME in path, f"The path must be a absolute path from `/{DOCKER_WORKPLACE_NAME}/` directory" + # local_path = path.replace('/' + DOCKER_WORKPLACE_NAME, LOCAL_ROOT + f'/{DOCKER_WORKPLACE_NAME}') + # print(local_path) + terminal_env.open_local_file(path) + header, content = _get_browser_state(terminal_env) + final_response = header.strip() + "\n==============================================\n" + content + "\n==============================================\n" + return final_response + except Exception as e: + return f"Error in `open_local_terminal_output`: {e}" + +@register_tool("terminal_page_up") +def terminal_page_up(): + """ + Scroll the viewport UP one page-length in the current terminal. Use this function when the terminal is too long and you want to scroll up to see the previous content. + """ + try: + terminal_env.page_up() + header, content = _get_browser_state(terminal_env) + final_response = header.strip() + "\n==============================================\n" + content + "\n==============================================\n" + return final_response + except Exception as e: + return f"Error in `page_up`: {e}" + +@register_tool("terminal_page_down") +def terminal_page_down(): + """ + Scroll the viewport DOWN one page-length in the current terminal. Use this function when the terminal is too long and you want to scroll down to see the next content. + """ + try: + terminal_env.page_down() + header, content = _get_browser_state(terminal_env) + final_response = header.strip() + "\n==============================================\n" + content + "\n==============================================\n" + return final_response + except Exception as e: + return f"Error in `page_down`: {e}" +@register_tool("terminal_page_to") +def terminal_page_to(page_idx: int): + """ + Move the viewport to the specified page index. The index starts from 1. + Use this function when you want to move the viewport to a specific page, especially when the middle of terminal output are meaningless, like the output of progress bar or output of generating directory structure when there are many datasets in the directory, you can use this function to move the viewport to the end of terminal where the meaningful content is. + """ + try: + terminal_env.page_to(page_idx - 1) + header, content = _get_browser_state(terminal_env) + final_response = header.strip() + "\n==============================================\n" + content + "\n==============================================\n" + return final_response + except Exception as e: + return f"Error in `page_to`: {e}" + +def process_terminal_agent_response(func): + """ + 装饰器函数，用于处理命令执行的响应结果 + - 如果结果是包含 status 和 result 的字典，返回格式化后的结果 + - 如果结果是错误字符串，直接返回 + """ + # original_func = func # 保存原始函数引用 + @wraps(func) # 保持原函数的签名和文档 + def wrapper(*args, **kwargs): + result = func(*args, **kwargs) + + # 如果返回值是字典且包含 status 和 result + if isinstance(result, dict) and 'status' in result and 'result' in result: + try: + res_output = result['result'] + if res_output == "": res_output = " " + if result['status'] != 0: + res_output = f"[ERROR] {res_output}" + else: + res_output = f"[SUCCESS] {res_output}" + tmp_file = os.path.join(os.getcwd(), "terminal_tmp", "terminal_output_{}___{}.txt".format(datetime.now().strftime("%Y%m%d_%H%M%S"), func.__name__)) + with open(tmp_file, "w") as f: + f.write(res_output) + return open_local_terminal_output(tmp_file) + except Exception as e: + return f"Error in the post-processing of `{func.__name__}`: {e}" + + elif isinstance(result, str): + return result + else: + return f"Error in `{func.__name__}`: {result}" + # 复制原始函数的签名到包装函数 + # 保持原始函数的属性 + return wrapper + +def process_terminal_response(func): + """ + 装饰器函数，用于处理命令执行的响应结果 + - 如果结果是包含 status 和 result 的字典，返回格式化后的结果 + - 如果结果是错误字符串，直接返回 + """ + # original_func = func # 保存原始函数引用 + @wraps(func) # 保持原函数的签名和文档 + def wrapper(*args, **kwargs): + result = func(*args, **kwargs) + + # 如果返回值是字典且包含 status 和 result + if isinstance(result, dict) and 'status' in result and 'result' in result: + try: + res_output = result['result'] + if res_output == "": res_output = " " + if result['status'] != 0: + res_output = f"[ERROR] {res_output}" + else: + res_output = f"[SUCCESS] {res_output}" + tmp_file = os.path.join(os.getcwd(), "terminal_tmp", "terminal_output_{}___{}.txt".format(datetime.now().strftime("%Y%m%d_%H%M%S"), func.__name__)) + Path(tmp_file).parent.mkdir(parents=True, exist_ok=True) + with open(tmp_file, "w") as f: + f.write(res_output) + return open_local_terminal_output(tmp_file) + except Exception as e: + return f"Error in the post-processing of `{func.__name__}`: {e}" + + elif isinstance(result, str): + return result + else: + return f"Error in `{func.__name__}`: {result}" + # 复制原始函数的签名到包装函数 + # 保持原始函数的属性 + return wrapper +@register_tool("read_file") +@process_terminal_response +def read_file(file_path: str, context_variables) -> str: + """ + Read the contents of a file and return it as a string. Use this function when there is a need to check an existing file. + Args: + file_path: The path of the file to read. + Returns: + A string representation of the contents of the file. + """ + env: Union[DockerEnv, LocalEnv] = context_variables.get("code_env", LocalEnv()) + try: + command = f"cat {file_path}" + response = env.run_command(command) # status, result + # res_output = truncate_by_tokens(env, response['result'], 10000) + # return f"Exit code: {response['status']} \nOutput: \n{res_output}" + return response + except FileNotFoundError: + return f"[ERROR] Error in reading file: {file_path}" + +def write_file_in_chunks(file_content, output_path, env: DockerEnv, chunk_size=100000): + encoded_content = base64.b64encode(file_content.encode('utf-8')).decode('utf-8') + total_chunks = math.ceil(len(encoded_content) / chunk_size) + + for i in range(total_chunks): + start = i * chunk_size + end = (i + 1) * chunk_size + chunk = encoded_content[start:end] + + # use cat command + if i == 0: + command = f"echo \"{chunk}\" | base64 -d > {output_path}" + else: + command = f"echo \"{chunk}\" | base64 -d >> {output_path}" + + response = env.run_command(command) + + if response["status"] != 0: + return f"Error creating file {output_path}: " + response["result"] + + # print(f"Successfully written block {i+1}/{total_chunks}") + + return f"File created at: {output_path}" + +@register_tool("create_file") +def create_file(path: str, content: str, context_variables) -> str: + """ + Create a file with the given path and content. Use this function when there is a need to create a new file with initial content. + Args: + path: The path to the file to create. + content: The initial content to write to the file. + Returns: + A string representation of the result of the file creation. + """ + env: Union[DockerEnv, LocalEnv] = context_variables.get("code_env", LocalEnv()) + try: + msg = write_file_in_chunks(content, path, env) + return msg + except Exception as e: + return f"Error creating file: {str(e)}" + +@register_tool("write_file") +def write_file(path: str, content: str, context_variables) -> str: + """ + Write content to a file. Use this function when there is a need to write content to an existing file. + Args: + path: The path to the file to write to. + content: The content to write to the file. + Returns: + A string representation of the result of the file writing. + """ + env: Union[DockerEnv, LocalEnv] = context_variables.get("code_env", LocalEnv()) + try: + msg = write_file_in_chunks(content, path, env) + return msg + except Exception as e: + return f"Error writing to file: {str(e)}" + +@register_tool("list_files") +@process_terminal_response +def list_files(path: str, context_variables) -> str: + """ + List all files and directories under the given path if it is a directory. Use this function when there is a need to list the contents of a directory. + Args: + path: The file system path to check and list contents from. + Returns: + A string representation of the contents of the directory. + """ + env: Union[DockerEnv, LocalEnv] = context_variables.get("code_env", LocalEnv()) + if os.path.isfile(path): + return "[ERROR] The given path is a file. Please provide a path of a directory." + command = f"ls -1 {path}" + response = env.run_command(command) + if response["status"] != 0: + return f"[ERROR] Error listing files: {response['result']}" + return response + +@register_tool("create_directory") +def create_directory(path: str, context_variables) -> str: + """ + Create a directory if it does not exist. Use this function when there is a need to create a new directory. + Args: + path: The path of the directory to create. + Returns: + A string representation of the result of the directory creation. + """ + env: Union[DockerEnv, LocalEnv] = context_variables.get("code_env", LocalEnv()) + try: + command = f"mkdir -p {path}" + response = env.run_command(command) + if response["status"] != 0: + return f"Error creating directory: {response['result']}" + return f"Directory '{path}' created successfully." + except OSError as error: + return f"Creation of the directory '{path}' failed due to: {error}" + +@register_tool("gen_code_tree_structure") +@process_terminal_response +def gen_code_tree_structure(directory: str, context_variables) -> str: + """Generate a tree structure of the code in the specified directory. Use this function when you need to know the overview of the codebase and want to generate a tree structure of the codebase. + Args: + directory: The directory to generate the tree structure for. + Returns: + A string representation of the tree structure of the code in the specified directory. + """ + env: Union[DockerEnv, LocalEnv] = context_variables.get("code_env", LocalEnv()) + try: + command = f"tree {directory}" + response = env.run_command(command) + return response + except Exception as e: + return f"[ERROR] Error running tree {directory}: {str(e)}" + +def print_stream(text): + console = Console() + console.print(f"[grey42]{text}[/grey42]") +@register_tool("execute_command") +@process_terminal_response +def execute_command(command: str, context_variables) -> str: + """ + Execute a command in the system shell. Use this function when there is a need to run a system command, and execute programs. + Args: + command: The command to execute in the system shell. + Returns: + A string representation of the exit code and output of the command. + """ + env: Union[DockerEnv, LocalEnv] = context_variables.get("code_env", LocalEnv()) + try: + response = env.run_command(command, print_stream) + return response + except Exception as e: + return f"[ERROR] Error running command: {str(e)}" + +def print_stream(text): + console = Console() + def escape_inner_tags(text): + # 先保护[grey42]标签 + text = text.replace("[grey42]", "###GREY42_START###") + text = text.replace("[/grey42]", "###GREY42_END###") + + # 转义所有其他的[]标签 + text = text.replace("[", "\\[") + text = text.replace("]", "\\]") + + # 恢复[grey42]标签 + text = text.replace("###GREY42_START###", "[grey42]") + text = text.replace("###GREY42_END###", "[/grey42]") + + return text + escaped_text = escape_inner_tags(text) + console.print(f"[grey42]{escaped_text}[/grey42]") +def set_doc(doc_template): + def decorator(func): + func.__doc__ = doc_template + return func + return decorator + +@register_tool("run_python") +@process_terminal_response +def run_python( + context_variables, + code_path: str, + cwd: str = None, + env_vars: Optional[Dict[str, str]] = None, +) -> str: + """ + Run a python script. + Args: + code_path: The absolute or relative path (the relative path is from the root of the workplace `/workplace`) to the python script file. + cwd: The working directory of the python script. If not provided, will regard the directory of the script as the working directory. If there is a command `cd ...` in the instruction for running the script, you should provide the cwd and not use the default value. (Optional) + env_vars: The environment variables to be set before running the python script. (Optional) + Returns: + A string representation of the exit code and output of the python script. + """ + env: Union[DockerEnv, LocalEnv] = context_variables.get("code_env", LocalEnv()) + try: + # 转换为绝对路径 + # abs_path = str(Path(code_path).resolve()) + if Path(code_path).is_absolute(): + if env.run_command(f"ls {code_path}")['status'] != 0: return f"[ERROR] File {code_path} does not exist" + code_abs_path = code_path + else: + code_abs_path = f"{env.docker_workplace}/{code_path}" + if env.run_command(f"ls {code_abs_path}")['status'] != 0: return f'[ERROR] You use a relative path, so we regard the `{env.docker_workplace}` as the root of the workplace, but `{code_abs_path}` does not exist' + + + if cwd: + # 使用指定的项目根目录 + if Path(cwd).is_absolute(): + if env.run_command(f"ls {cwd}")['status'] != 0: return f"[ERROR] Working directory {cwd} does not exist" + else: + cwd = f"{env.docker_workplace}/{cwd}" + if env.run_command(f"ls {cwd}")['status'] != 0: return f"[ERROR] You use a relative path for `cwd`, so we regard the `{env.docker_workplace}` as the working directory, but `{cwd}` does not exist" + else: + cwd = str(Path(code_abs_path).parent) + + + # 设置PYTHONPATH + pythonpath = str(cwd) + + # 获取Python解释器路径 + env_str = f"PYTHONPATH={pythonpath}" + + if env_vars: + env_str += " " + " ".join([f"{k}={v}" for k, v in env_vars.items()]) + # print(env_str) + + # 构建相对模块路径 + try: + rel_path = Path(code_abs_path).relative_to(cwd) + module_path = str(rel_path.with_suffix('')).replace(os.sep, '.') + + command = f"cd {cwd} && {env_str} python -m {module_path}" + except ValueError: + # 如果无法构建相对路径，使用完整路径 + command = f"cd {cwd} && {env_str} python {code_path}" + + # print(f"Executing: {command}") + + result = env.run_command(command, print_stream) + return result + + except Exception as e: + return f"[ERROR] Error when running the python script: {e}" + + +if __name__ == "__main__": + env_config = DockerConfig( + container_name = "paper_eval_dit", + workplace_name = "workplace", + communication_port = 12347, + conda_path = "/home/user/micromamba", + local_root = "/home/tjb/llm/agent/Inno-agent/workplace_paper/task_dit/workplace" + ) + env = DockerEnv(env_config) + + # print(read_file("/workplace/lucidrains_denoising_diffusion/denoising_diffusion_pytorch/denoising_diffusion_pytorch.py", env)) + # print(terminal_page_to(3)) + sig = inspect.signature(execute_command) + print("Parameters from signature:", list(sig.parameters.keys())) + # print(terminal_page_down()) + # print(terminal_page_down()) + # print(terminal_page_down()) + # print(terminal_page_down()) + # print(execute_command("cp project/configs.py ./", env)) + diff --git a/autoagent/tools/tool_utils.py b/autoagent/tools/tool_utils.py new file mode 100644 index 0000000..acff648 --- /dev/null +++ b/autoagent/tools/tool_utils.py @@ -0,0 +1,24 @@ +from autoagent.environment.docker_env import DockerEnv + +import tiktoken +from datetime import datetime + +def truncate_by_tokens(env: DockerEnv, text, max_tokens = 4096, model="gpt-4o-2024-08-06"): + from autoagent.tools.terminal_tools import create_file + encoding = tiktoken.encoding_for_model(model) + tokens = encoding.encode(text) + + if len(tokens) <= max_tokens: + return text + timestamp = datetime.now().strftime("%Y%m%d_%H%M%S") + + output_path = f"{env.docker_workplace}/console_output/truncated_output_{timestamp}.txt" + create_msg = create_file(output_path, content = text, context_variables={'code_env': env}) + # 截断tokens并解码回字符串 + truncated_tokens_bos = tokens[:max_tokens//2] + truncated_tokens_eos = tokens[-(max_tokens - len(truncated_tokens_bos)):] + if create_msg.startswith("Error"): + return encoding.decode(truncated_tokens_bos) + "\n...\n" + encoding.decode(truncated_tokens_eos) + "\n\nThe full console output is too long, so I want to save them into the file: {0}\n\nBut I got an error: {1}".format(output_path, create_msg) + + return encoding.decode(truncated_tokens_bos) + "\n...\n" + encoding.decode(truncated_tokens_eos) + "\n\nThe full console output is too long, so it is saved in the file: {0}\n\nYou may use the `File Surfer Agent` to view the full output.".format(output_path) + diff --git a/autoagent/tools/web_tools.py b/autoagent/tools/web_tools.py new file mode 100644 index 0000000..625b8ff --- /dev/null +++ b/autoagent/tools/web_tools.py @@ -0,0 +1,462 @@ +from autoagent.registry import register_tool +from browsergym.core.action.highlevel import HighLevelActionSet +from typing import Literal +from autoagent.environment import DockerEnv, DockerConfig, BrowserEnv, VIEWPORT +from browsergym.utils.obs import flatten_axtree_to_str +from dataclasses import dataclass, field +from typing import Dict +from urllib.parse import quote_plus +from autoagent.types import Result +from functools import partial, update_wrapper +from inspect import signature +import tiktoken +from datetime import datetime +from autoagent.util import function_to_json +from autoagent.tools.md_obs import flatten_md_axtree_to_str +# def with_env(env: BrowserEnv): +# """将env注入到工具函数中的装饰器""" +# def decorator(func): +# # 创建新函数，固定env参数 +# new_func = partial(func, env=env) +# # 保留原始函数的docstring和signature +# update_wrapper(new_func, func) +# # 修改signature，移除env参数 +# new_func.__signature__ = signature(func).replace( +# parameters=[p for p in signature(func).parameters.values() if p.name != 'env'] +# ) +# return new_func +# return decorator +def with_env(env: BrowserEnv): + """将env注入到工具函数中的装饰器""" + def decorator(func): + def wrapped(*args, **kwargs): + return func(env=env, *args, **kwargs) + + # 保留原始函数的所有属性 + update_wrapper(wrapped, func) + # 修改signature，移除env参数 + wrapped.__signature__ = signature(func).replace( + parameters=[p for p in signature(func).parameters.values() if p.name != 'env'] + ) + return wrapped + return decorator + +def with_two_envs(env: BrowserEnv, code_env: DockerEnv): + """将env注入到工具函数中的装饰器""" + def decorator(func): + def wrapped(*args, **kwargs): + return func(env=env, code_env=code_env, *args, **kwargs) + + # 保留原始函数的所有属性 + update_wrapper(wrapped, func) + # 修改signature，移除env参数 + wrapped.__signature__ = signature(func).replace( + parameters=[p for p in signature(func).parameters.values() if p.name not in ['env', 'code_env']] + ) + return wrapped + return decorator +@dataclass +class WebObservation: + content: str # text content of the page + url: str # URL of the page + screenshot: str # base64-encoded screenshot, png + open_pages_urls: list[str] # list of open pages + active_page_index: int # index of the active page + dom_object: dict # DOM object + axtree_object: dict # accessibility tree object + extra_element_properties: dict + focused_element_bid: str # focused element bid + last_browser_action: str # last browser env action performed + last_browser_action_error: str # last browser env action error + error: bool # error flag + +def to_web_obs(obs) -> WebObservation: + obs_dict = dict( + content=obs['text_content'], # text content of the page + url=obs.get('url', ''), # URL of the page + # screenshot=obs.get('screenshot', None), # base64-encoded screenshot, png + screenshot=None, # base64-encoded screenshot, png + open_pages_urls=obs.get('open_pages_urls', []), # list of open pages + active_page_index=obs.get( + 'active_page_index', -1 + ), # index of the active page + dom_object=obs.get('dom_object', {}), # DOM object + axtree_object=obs.get('axtree_object', {}), # accessibility tree object + extra_element_properties=obs.get('extra_element_properties', {}), + focused_element_bid=obs.get( + 'focused_element_bid', None + ), # focused element bid + last_browser_action=obs.get( + 'last_action', '' + ), # last browser env action performed + last_browser_action_error=obs.get('last_action_error', ''), + error=True if obs.get('last_action_error', '') else False, # error flag + ) + return WebObservation(**obs_dict) +def wrap_return_value(web_obs: WebObservation, action_description: str = ""): + error_prefix = "" + if web_obs.error: + error_prefix = get_error_prefix(web_obs.last_browser_action, web_obs.last_browser_action_error) + cur_url = web_obs.url + try: + cur_axtree_txt = flatten_axtree_to_str( + web_obs.axtree_object, + extra_properties=web_obs.extra_element_properties, + with_clickable=True, + filter_visible_only=True, + ) + except Exception as e: + cur_axtree_txt = f'Error encountered when browsing.\nError when trying to process the accessibility tree:{str(e)}' + ret_value = f"""\ +{error_prefix} +{action_description} + +# Current Page URL: +{cur_url} + +# Current Accessibility Tree: +{cur_axtree_txt} + +Here is an example with chain of thought of a valid action when clicking on a button: +" +In order to accomplish my goal I need to click on the button with bid 12 +```click("12")``` +" +""".strip() + return ret_value + +def wrap_return_value_markdown(web_obs: WebObservation, action_description: str = ""): + error_prefix = "" + if web_obs.error: + error_prefix = get_error_prefix(web_obs.last_browser_action, web_obs.last_browser_action_error) + cur_url = web_obs.url + try: + cur_axtree_txt = flatten_md_axtree_to_str( + web_obs.axtree_object, + extra_properties=web_obs.extra_element_properties, + with_clickable=True, + filter_visible_only=True, + ) + except Exception as e: + cur_axtree_txt = f'Error encountered when browsing.\nError when trying to process the accessibility tree:{str(e)}' + ret_value = f"""\ +{error_prefix} +{action_description} + +# The current page is converted to markdown format: +{cur_axtree_txt} + +If the content is too long, you can use `page_down()` and `page_up()` to navigate through the text. +If you have not yet got the answer and want to back to the previous page, please use `history_back()` to navigate back. +""".strip() + return ret_value +def get_error_prefix(last_browser_action: str, last_browser_action_error: str) -> str: + return f'IMPORTANT! Last action is incorrect:\n{last_browser_action}\nThink again with the current observation of the page.\nThe error message is:\n{last_browser_action_error}' + +# @register_tool("click") +# def click(env: BrowserEnv, bid: str, button: Literal["left", "middle", "right"] = "left", modifiers: list[Literal["Alt", "Control", "ControlOrMeta", "Meta", "Shift"]] = []): +# """ +# Clicks the mouse on the target with the given element bid. +# Args: +# bid: The bid of the element to click. +# button: The button to click. +# modifiers: The modifiers to click. +# """ +# try: +# # 执行动作 +# # action = action_func(*args, **kwargs) +# button_str = f''', button="{button}"''' if button else '' +# modifiers_str = f', modifiers={modifiers}' if modifiers else '' +# action_str = f"""click('{bid}'{button_str}{modifiers_str})""" + +# # 与环境交互 +# obs = env.step(action_str) +# web_obs = to_web_obs(obs) + +# except Exception as e: +# return f"Error encountered when taking action: {action_str}\nError: {e}" +# ret_value = wrap_return_value(web_obs) +# return Result( +# value=ret_value, +# image=web_obs.screenshot, +# ) +@register_tool("click") +def click(context_variables, bid: str, button: Literal["left", "middle", "right"] = "left"): + """ + Clicks the mouse on the target with the given element bid. + Args: + bid: The bid of the element to click. + button: The button to click. + """ + env: BrowserEnv = context_variables.get("web_env", None) + assert env is not None, "web_env is not set" + try: + # 执行动作 + # action = action_func(*args, **kwargs) + button_str = f''', button="{button}"''' if button else '' + action_str = f"""_click_id('{bid}'{button_str})""" + + # 与环境交互 + obs = env.step(action_str) + web_obs = to_web_obs(obs) + + except Exception as e: + return f"Error encountered when taking action: {action_str}\nError: {e}" + ret_value = wrap_return_value(web_obs) + return Result( + value=ret_value, + image=web_obs.screenshot, + ) +@register_tool("page_down") +def page_down(context_variables): + """ + Scrolls the entire browser viewport one page DOWN towards the end. + """ + env: BrowserEnv = context_variables.get("web_env", None) + assert env is not None, "web_env is not set" + try: + action_str = f'scroll(0, {VIEWPORT["height"]-50})' + obs = env.step(action_str) + web_obs = to_web_obs(obs) + except Exception as e: + return f"Error encountered when taking action: {action_str}\nError: {e}" + if web_obs.url.startswith("data:text/html;base64,"): + ret_value = wrap_return_value_markdown(web_obs) + else: + ret_value = wrap_return_value(web_obs) + + return Result( + value=ret_value, + image=web_obs.screenshot, + ) +@register_tool("page_up") +def page_up(context_variables): + """ + Scrolls the entire browser viewport one page UP towards the beginning. + """ + env: BrowserEnv = context_variables.get("web_env", None) + assert env is not None, "web_env is not set" + try: + action_str = f'scroll(0, -{VIEWPORT["height"]-50})' + obs = env.step(action_str) + web_obs = to_web_obs(obs) + except Exception as e: + return f"Error encountered when taking action: {action_str}\nError: {e}" + if web_obs.url.startswith("data:text/html;base64,"): + ret_value = wrap_return_value_markdown(web_obs) + else: + ret_value = wrap_return_value(web_obs) + return Result( + value=ret_value, + image=web_obs.screenshot, + ) +@register_tool("history_back") +def history_back(context_variables): + """ + Navigates back one page in the browser's history. This is equivalent to clicking the browser back button. + """ + env: BrowserEnv = context_variables.get("web_env", None) + assert env is not None, "web_env is not set" + try: + action_str = 'go_back()' + obs = env.step(action_str) + web_obs = to_web_obs(obs) + except Exception as e: + return f"Error encountered when taking action: {action_str}\nError: {e}" + ret_value = wrap_return_value(web_obs) + return Result( + value=ret_value, + image=web_obs.screenshot, + ) +@register_tool("history_forward") +def history_forward(context_variables): + """ + Navigates forward one page in the browser's history. This is equivalent to clicking the browser forward button. + """ + env: BrowserEnv = context_variables.get("web_env", None) + assert env is not None, "web_env is not set" + try: + action_str = 'go_forward()' + obs = env.step(action_str) + web_obs = to_web_obs(obs) + except Exception as e: + return f"Error encountered when taking action: {action_str}\nError: {e}" + ret_value = wrap_return_value(web_obs) + return Result( + value=ret_value, + image=web_obs.screenshot, + ) +@register_tool("input_text") +def input_text(context_variables, bid: str, text: str): + """ + Types the given text value into the specified field. + Args: + bid: The bid of the element to type into. + text: The text to type into the input field. + """ + env: BrowserEnv = context_variables.get("web_env", None) + assert env is not None, "web_env is not set" + try: + action_str = f"fill('{bid}', '{text}')" + obs = env.step(action_str) + web_obs = to_web_obs(obs) + except Exception as e: + return f"Error encountered when taking action: {action_str}\nError: {e}" + ret_value = wrap_return_value(web_obs) + return Result( + value=ret_value, + image=web_obs.screenshot, + ) + +@register_tool("visit_url") +def visit_url(context_variables, url: str): + """ + Navigate directly to a provided URL using the browser's address bar. Prefer this tool over other navigation techniques in cases where the user provides a fully-qualified URL (e.g., choose it over clicking links, or inputing queries into search boxes). + Args: + url: The URL to navigate to. + """ + env: BrowserEnv = context_variables.get("web_env", None) + assert env is not None, "web_env is not set" + try: + if url.startswith(("https://", "http://", "file://", "about:")): + action_str = f"_visit_page('{url}')" + obs = env.step(action_str) + web_obs = to_web_obs(obs) + elif " " in url: + query = quote_plus(url) + action_str = f"_visit_page('https://www.google.com.sg/search?q={query}&hl=en&gl=US')" + obs = env.step(action_str) + web_obs = to_web_obs(obs) + else: + action_str = f"_visit_page('https://{url}')" + obs = env.step(action_str) + web_obs = to_web_obs(obs) + except Exception as e: + return f"Error encountered when taking action: {action_str}\nError: {e}" + ret_value = wrap_return_value(web_obs) + return Result( + value=ret_value, + image=web_obs.screenshot, + ) + +@register_tool("web_search") +def web_search(context_variables, query: str): + """ + Performs a web search on 'https://www.bing.com/search' with the given query. + Args: + query: The query to search for. + """ + env: BrowserEnv = context_variables.get("web_env", None) + assert env is not None, "web_env is not set" + try: + # action_str = f"_visit_page('https://www.google.com.sg/search?q={quote_plus(query)}&hl=en')" + action_str = f"_visit_page('https://www.bing.com/search?q={quote_plus(query)}&FORM=QBLH&hl=en')" + + obs = env.step(action_str) + web_obs = to_web_obs(obs) + except Exception as e: + return f"Error encountered when taking action: {action_str}\nError: {e}" + ret_value = wrap_return_value(web_obs) + return Result( + value=ret_value, + image=web_obs.screenshot, + ) +@register_tool("sleep") +def sleep(context_variables): + """ + Wait a short period of time. Call this function if the page has not yet fully loaded, or if it is determined that a small delay would increase the task's chances of success. + """ + env: BrowserEnv = context_variables.get("web_env", None) + assert env is not None, "web_env is not set" + + try: + action_str = f"noop(3000)" + obs = env.step(action_str) + web_obs = to_web_obs(obs) + except Exception as e: + return f"Error encountered when taking action: {action_str}\nError: {e}" + ret_value = wrap_return_value(web_obs) + return Result( + value=ret_value, + image=web_obs.screenshot, + ) +def truncate_by_tokens(env: DockerEnv, text, max_tokens = 4096, model="gpt-4o-2024-08-06"): + from autoagent.tools.terminal_tools import create_file, create_directory + encoding = tiktoken.encoding_for_model(model) + tokens = encoding.encode(text) + + if len(tokens) <= max_tokens: + return text + timestamp = datetime.now().strftime("%Y%m%d_%H%M%S") + create_directory(f"{env.docker_workplace}/web_page_markdown_output", context_variables={"code_env": env}) + output_path = f"{env.docker_workplace}/web_page_markdown_output/full_output_{timestamp}.md" + create_msg = create_file(output_path, content = text, context_variables={"code_env": env}) + # 截断tokens并解码回字符串 + truncated_tokens_bos = tokens[:max_tokens//2] + truncated_tokens_eos = tokens[-(max_tokens - len(truncated_tokens_bos)):] + if create_msg.startswith("Error"): + return encoding.decode(truncated_tokens_bos) + "\n...\n" + encoding.decode(truncated_tokens_eos) + "\n\nThe full convert markdown output is too long, so I want to save them into the file: {0}\n\nBut I got an error: {1}".format(output_path, create_msg) + + return encoding.decode(truncated_tokens_bos) + "\n...\n" + encoding.decode(truncated_tokens_eos) + "\n\nThe full convert markdown output is too long, so it is saved in the file: {0}\n\nYou may use the `File Surfer Agent` to view the full output.".format(output_path) + +@register_tool("get_page_markdown") +def get_page_markdown(context_variables): + """ + Get the markdown content of the current page. + Use this tool if you need to watch the Youtube video, Wikipedia page, or other pages that contain media content. + Note that this tool can only be used after you have visited a valid page. + """ + env: BrowserEnv = context_variables.get("web_env", None) + assert env is not None, "web_env is not set" + # code_env: DockerEnv = context_variables.get("code_env", None) + # assert code_env is not None, "code_env is not set" + try: + action_str = "_get_page_markdown()" + obs = env.step(action_str) + web_obs = to_web_obs(obs) + # obs = env.step("go_back()") + except Exception as e: + return f"Error encountered when taking action: {action_str}\nError: {e}" + +# ret_value = \ +# f""" +# I have converted the current page into clean markdown format: +# {web_obs.content} +# """.strip() + ret_value = wrap_return_value_markdown(web_obs) + # ret_value = truncate_by_tokens(code_env, ret_value, max_tokens=10000) + return Result( + value=ret_value, + image=web_obs.screenshot, + ) + +if __name__ == "__main__": + env = BrowserEnv(browsergym_eval_env = None, local_root="/Users/tangjiabin/Documents/reasoning/autoagent", workplace_name="workplace_gaia_eval") + # code_env = DockerEnv(DockerConfig(container_name = "gaia_lite_eval", + # workplace_name = "workplace_gaia_eval", + # communication_port = 12345, + # conda_path = "/root/miniconda3")) + # code_env.init_container() + # import json + # web_search_with_env = with_env(env)(web_search) + # print(json.dumps(function_to_json(web_search_with_env), indent=4)) + # visit_url(env, "https://scholar.google.com.hk/scholar?hl=zh-CN&as_sdt=0%2C5&q=LLMRec&oq=") + # res = page_down(env) + # print(res.value) + # res = visit_url(env, 'https://arxiv.org/pdf/2310.13023') + # print(res.value) + + + context_variables = {"web_env": env} + res = visit_url(context_variables, 'https://www.youtube.com/watch?v=pPStdjuYzSI') + # res = visit_url(env, 'https://www.reddit.com/r/ChatGPT/comments/1h5ey4m/chatgpt_helped_me_not_blow_up_on_my_boss/') + print("******visit_url", res.value) + + res = get_page_markdown(context_variables) + print("******get_page_markdown", res.value) + # res = page_down(context_variables) + # print("******page_down", res.value) + # res = history_back(context_variables) + # print("******history_back", res.value) + + # print(function_to_json(page_down)) \ No newline at end of file diff --git a/autoagent/types.py b/autoagent/types.py new file mode 100644 index 0000000..500c6f5 --- /dev/null +++ b/autoagent/types.py @@ -0,0 +1,41 @@ +from litellm.types.utils import ChatCompletionMessageToolCall, Function, Message +from typing import List, Callable, Union, Optional, Tuple, Dict + +# Third-party imports +from pydantic import BaseModel + +AgentFunction = Callable[[], Union[str, "Agent", dict]] + + +class Agent(BaseModel): + name: str = "Agent" + model: str = "gpt-4o" + instructions: Union[str, Callable[[], str]] = "You are a helpful agent." + functions: List[AgentFunction] = [] + tool_choice: str = None + parallel_tool_calls: bool = False + examples: Union[List[Tuple[dict, str]], Callable[[], str]] = [] + handle_mm_func: Callable[[], str] = None + agent_teams: Dict[str, Callable] = {} + + +class Response(BaseModel): + messages: List = [] + agent: Optional[Agent] = None + context_variables: dict = {} + + +class Result(BaseModel): + """ + Encapsulates the possible return values for an agent function. + + Attributes: + value (str): The result value as a string. + agent (Agent): The agent instance, if applicable. + context_variables (dict): A dictionary of context variables. + """ + + value: str = "" + agent: Optional[Agent] = None + context_variables: dict = {} + image: Optional[str] = None # base64 encoded image diff --git a/autoagent/util.py b/autoagent/util.py new file mode 100644 index 0000000..fc454ff --- /dev/null +++ b/autoagent/util.py @@ -0,0 +1,540 @@ +import inspect +from datetime import datetime +import socket +import json +import uuid +from typing import Callable, List, Dict, Any, Optional, Callable, Union, get_args, get_origin +from dataclasses import is_dataclass, fields, MISSING +from pydantic import BaseModel +from rich.panel import Panel +from rich.prompt import Prompt +from rich.console import Console +import inquirer +from rich.markdown import Markdown +from prompt_toolkit import PromptSession +from prompt_toolkit.completion import Completer, Completion +from prompt_toolkit.formatted_text import HTML +from prompt_toolkit.styles import Style +def debug_print_swarm(debug: bool, *args: str) -> None: + if not debug: + return + timestamp = datetime.now().strftime("%Y-%m-%d %H:%M:%S") + message = " ".join(map(str, args)) + print(f"\033[97m[\033[90m{timestamp}\033[97m]\033[90m {message}\033[0m") +def print_in_box(text: str, console: Optional[Console] = None, title: str = "", color: str = "white") -> None: + """ + Print the text in a box. + :param text: the text to print. + :param console: the console to print the text. + :param title: the title of the box. + :param color: the border color. + :return: + """ + console = console or Console() + + # panel = Panel(text, title=title, border_style=color, expand=True, highlight=True) + # console.print(panel) + console.print('_'*20 + title + '_'*20, style=f"bold {color}") + console.print(text, highlight=True, emoji=True) + + + +def debug_print(debug: bool, *args: str, **kwargs: dict) -> None: + if not debug: + return + timestamp = datetime.now().strftime("%Y-%m-%d %H:%M:%S") + message = "\n".join(map(str, args)) + color = kwargs.get("color", "white") + title = kwargs.get("title", "") + log_str = f"[{timestamp}]\n{message}" + print_in_box(log_str, color=color, title=title) + log_path = kwargs.get("log_path", None) + if log_path: + with open(log_path, 'a') as f: + f.write(log_str + '\n') + + + +def ask_text(question: str, title: str = "User", console: Optional[Console] = None, default_answer: str = "") -> str: + """ + Display a question in a panel and prompt the user for an answer. + :param question: the question to display. + :param title: the title of the panel. + :param console: the console to use. + :return: the user's answer. + """ + console = console or Console() + + console.print(Panel(question, title=title, border_style="green")) + answer = Prompt.ask(f"Type your answer here, press Enter to use default answer", default=default_answer) + console.print(Panel(answer, title=title)) + return answer + +def print_markdown(md_path: str, console: Optional[Console] = None): + console = console or Console() + with open(md_path, 'r') as f: + md_content = f.read() + console.print(Markdown(md_content)) + +def single_select_menu(options, message: str = ""): + questions = [ + inquirer.List( + 'choice', + message=message, + choices=options, + ), + ] + answers = inquirer.prompt(questions) + return answers['choice'] + + +def get_user_confirmation(prompt: str) -> bool: + user_input = prompt.strip().lower() + if user_input in ['y', 'yes', 'true', 't']: + return True + elif user_input in ['n', 'no', 'false', 'f'] or user_input == '': + return False + else: + print("Invalid input. Please enter 'y' for yes or 'n' for no.") + +def merge_fields(target, source): + for key, value in source.items(): + if isinstance(value, str): + target[key] += value + elif value is not None and isinstance(value, dict): + merge_fields(target[key], value) + + +def merge_chunk(final_response: dict, delta: dict) -> None: + delta.pop("role", None) + merge_fields(final_response, delta) + + tool_calls = delta.get("tool_calls") + if tool_calls and len(tool_calls) > 0: + index = tool_calls[0].pop("index") + merge_fields(final_response["tool_calls"][index], tool_calls[0]) + + +# def function_to_json(func) -> dict: +# """ +# Converts a Python function into a JSON-serializable dictionary +# that describes the function's signature, including its name, +# description, and parameters. + +# Args: +# func: The function to be converted. + +# Returns: +# A dictionary representing the function's signature in JSON format. +# """ +# type_map = { +# str: "string", +# int: "integer", +# float: "number", +# bool: "boolean", +# list: "array", +# dict: "object", +# type(None): "null", +# } + +# try: +# signature = inspect.signature(func) +# except ValueError as e: +# raise ValueError( +# f"Failed to get signature for function {func.__name__}: {str(e)}" +# ) + +# parameters = {} +# for param in signature.parameters.values(): +# try: +# param_type = type_map.get(param.annotation, "string") +# except KeyError as e: +# raise KeyError( +# f"Unknown type annotation {param.annotation} for parameter {param.name}: {str(e)}" +# ) +# parameters[param.name] = {"type": param_type} + +# required = [ +# param.name +# for param in signature.parameters.values() +# if param.default == inspect._empty +# ] + +# return { +# "type": "function", +# "function": { +# "name": func.__name__, +# "description": func.__doc__ or "", +# "parameters": { +# "type": "object", +# "properties": parameters, +# "required": required, +# }, +# }, +# } + +def get_type_info(annotation, base_type_map): + # 处理基本类型 + if annotation in base_type_map: + return {"type": base_type_map[annotation]} + + # 处理typing类型 + origin = get_origin(annotation) + if origin is not None: + args = get_args(annotation) + + # 处理List类型 + if origin is list or origin is List: + item_type = args[0] + return { + "type": "array", + "items": get_type_info(item_type, base_type_map) + } + + # 处理Dict类型 + elif origin is dict or origin is Dict: + key_type, value_type = args + if key_type != str: + raise ValueError("Dictionary keys must be strings") + + # 如果value_type是TypedDict或Pydantic模型 + if (hasattr(value_type, "__annotations__") or + (isinstance(value_type, type) and issubclass(value_type, BaseModel))): + return get_type_info(value_type, base_type_map) + + # 普通Dict类型 + return { + "type": "object", + "additionalProperties": get_type_info(value_type, base_type_map) + } + + # 处理Union类型 + elif origin is Union: + types = [get_type_info(arg, base_type_map) for arg in args if arg != type(None)] + if len(types) == 1: + return types[0] + return {"oneOf": types} + + # 处理Pydantic模型 + if isinstance(annotation, type): + try: + if issubclass(annotation, BaseModel): + schema = annotation.model_json_schema() + # 提取主要schema部分 + properties = schema.get("properties", {}) + required = schema.get("required", []) + + # 处理definitions + definitions = schema.get("$defs", {}) + if definitions: + # 如果有引用的定义，直接展开它们 + for prop_name, prop_schema in properties.items(): + if "$ref" in prop_schema: + ref_name = prop_schema["$ref"].split("/")[-1] + if ref_name in definitions: + properties[prop_name] = definitions[ref_name] + + return { + "type": "object", + "properties": properties, + "required": required, + "additionalProperties": False + } + except TypeError: + pass + + # 处理dataclass + if is_dataclass(annotation): + properties = {} + required = [] + for field in fields(annotation): + properties[field.name] = get_type_info(field.type, base_type_map) + if field.default == field.default_factory == MISSING: + required.append(field.name) + + return { + "type": "object", + "properties": properties, + "required": required, + "additionalProperties": False + } + + # 处理TypedDict + if hasattr(annotation, "__annotations__"): + properties = {} + required = getattr(annotation, "__required_keys__", annotation.__annotations__.keys()) + + for key, field_type in annotation.__annotations__.items(): + properties[key] = get_type_info(field_type, base_type_map) + + return { + "type": "object", + "properties": properties, + "required": list(required), + "additionalProperties": False + } + + # 默认返回string类型 + return {"type": "string"} + + +def function_to_json(func) -> dict: + """ + Converts a Python function into a JSON-serializable dictionary + that describes the function's signature, including its name, + description, and parameters. + + Args: + func: The function to be converted. + + Returns: + A dictionary representing the function's signature in JSON format. + """ + type_map = { + str: "string", + int: "integer", + float: "number", + bool: "boolean", + # list: "array", + # dict: "object", + type(None): "null", + } + # def get_type_info(annotation): + # if hasattr(annotation, "__origin__"): # 处理typing类型 + # origin = annotation.__origin__ + # if origin is list: # 处理List类型 + # item_type = annotation.__args__[0] + # return { + # "type": "array", + # "items": { + # "type": type_map.get(item_type, "string") + # } + # } + # elif origin is dict: # 处理Dict类型 + # return {"type": "object"} + # return {"type": type_map.get(annotation, "string")} + + try: + signature = inspect.signature(func) + except ValueError as e: + raise ValueError( + f"Failed to get signature for function {func.__name__}: {str(e)}" + ) + + parameters = {} + # for param in signature.parameters.values(): + # try: + # param_type = type_map.get(param.annotation, "string") + # except KeyError as e: + # raise KeyError( + # f"Unknown type annotation {param.annotation} for parameter {param.name}: {str(e)}" + # ) + # parameters[param.name] = {"type": param_type} + for param in signature.parameters.values(): + if param.name == "context_variables": + continue + try: + param_info = get_type_info(param.annotation, type_map) + if isinstance(param_info, dict) and "additionalProperties" in param_info: + del param_info["additionalProperties"] + parameters[param.name] = get_type_info(param.annotation, type_map) + except KeyError as e: + raise KeyError(f"Unknown type annotation {param.annotation} for parameter {param.name}: {str(e)}") + + + + required = [ + param.name + for param in signature.parameters.values() + if param.default == inspect._empty + ] + + # if not parameters: + # parameters["dummy"] = { + # "type": "string", + # "description": "Dummy parameter (not used). Added to satisfy non-empty schema requirements." + # } + # required = [] + + return { + "type": "function", + "function": { + "name": func.__name__, + "description": func.__doc__ or "", + "parameters": { + "type": "object", + "properties": parameters, + "required": required, + }, + }, + } + +def run_command_in_container_v1(command, stream_callback: Callable = None): + # TCP parameters + hostname = 'localhost' + port = 12345 # TCP port mapped to the container + buffer_size = 4096 + + # Create TCP client + with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s: + s.connect((hostname, port)) + s.sendall(command.encode()) + full_response = b"" + while True: + chunk = s.recv(buffer_size) + if not chunk: + break + full_response += chunk + if stream_callback: + stream_callback(chunk) + if len(chunk) < buffer_size: + # If the received data is less than the buffer size, it may have been received + break + + # Decode the complete response + try: + decoded_response = full_response.decode('utf-8') + return json.loads(decoded_response) + except json.JSONDecodeError as e: + print(f"JSON parsing error: {e}") + print(f"Raw response received: {decoded_response}") + return {"status": -1, "result": "Response parsing error"} + +def run_command_in_container(command, stream_callback=None): + """ + communicate with docker container and execute command, support stream output + + Args: + command: the command to execute + stream_callback: optional callback function, for handling stream output + the function signature should be callback(text: str) + + Returns: + dict: the complete JSON result returned by the docker container + """ + hostname = 'localhost' + port = 12345 + buffer_size = 4096 + + with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s: + s.connect((hostname, port)) + s.sendall(command.encode()) + + partial_line = "" + while True: + chunk = s.recv(buffer_size) + if not chunk: + break + + # add new received data to the unfinished data + data = partial_line + chunk.decode('utf-8') + lines = data.split('\n') + + # except the last line, process all complete lines + for line in lines[:-1]: + if line: + try: + response = json.loads(line) + if response['type'] == 'chunk': + # process stream output + if stream_callback: + stream_callback(response['data']) + elif response['type'] == 'final': + # return the final result + return { + 'status': response['status'], + 'result': response['result'] + } + except json.JSONDecodeError: + print(f"Invalid JSON: {line}") + + # save the possibly unfinished last line + partial_line = lines[-1] + + # if the loop ends normally without receiving a final response + return { + 'status': -1, + 'result': 'Connection closed without final response' + } + + +def make_tool_message(tools: Callable, args: dict, tool_content: str) -> List[Dict]: + tool_calls = [ + { + "type": "function", + "function": { + "name": tools.__name__, + "arguments": json.dumps(args) + }, + "id": str(uuid.uuid4()).replace('-', '')[:9] + } + ] + return [ + {'role': 'assistant', 'tool_calls': tool_calls}, + {'role': 'tool', 'content': tool_content, 'name': tools.__name__, 'tool_call_id': tool_calls[0]['id']} + ] +def make_message(role: str, content: str): + return [ + {'role': role, 'content': content} + ] + + + + +class UserCompleter(Completer): + + def __init__(self, users: List[str]): + super().__init__() + self.users = users + def get_completions(self, document, complete_event): + word = document.get_word_before_cursor() + + if word.startswith('@'): + prefix = word[1:] # 去掉@ + for user in self.users: + if user.startswith(prefix): + yield Completion( + user, + start_position=-len(prefix), + style='fg:blue bold' # 蓝色加粗 + ) +def pretty_print_messages(message, **kwargs) -> None: + # for message in messages: + if message["role"] != "assistant" and message["role"] != "tool": + return + console = Console() + if message["role"] == "tool": + console.print("[bold blue]tool execution:[/bold blue]", end=" ") + console.print(f"[bold purple]{message['name']}[/bold purple], result: {message['content']}") + log_path = kwargs.get("log_path", None) + if log_path: + with open(log_path, 'a') as file: + file.write(f"tool execution: {message['name']}, result: {message['content']}\n") + return + + # print agent name in blue + console.print(f"[bold blue]{message['sender']}[/bold blue]:", end=" ") + + # print response, if any + if message["content"]: + console.print(message["content"], highlight=True, emoji=True) + + # print tool calls in purple, if any + tool_calls = message.get("tool_calls") or [] + if len(tool_calls) > 1: + console.print() + for tool_call in tool_calls: + f = tool_call["function"] + name, args = f["name"], f["arguments"] + arg_str = json.dumps(json.loads(args)).replace(":", "=") + console.print(f"[bold purple]{name}[/bold purple]({arg_str[1:-1]})") + log_path = kwargs.get("log_path", None) + if log_path: + with open(log_path, 'a') as file: + file.write(f"{message['sender']}: {message['content']}\n") + for tool_call in tool_calls: + f = tool_call["function"] + name, args = f["name"], f["arguments"] + arg_str = json.dumps(json.loads(args)).replace(":", "=") + file.write(f"{name}({arg_str[1:-1]})\n") + diff --git a/autoagent/workflows/__init__.py b/autoagent/workflows/__init__.py new file mode 100644 index 0000000..38f7b67 --- /dev/null +++ b/autoagent/workflows/__init__.py @@ -0,0 +1,39 @@ +import os +import importlib +from autoagent.registry import registry + +def import_workflows_recursively(base_dir: str, base_package: str): + """Recursively import all workflows in .py files + + Args: + base_dir: the root directory to start searching + base_package: the base name of the Python package + """ + for root, dirs, files in os.walk(base_dir): + # get the relative path to the base directory + rel_path = os.path.relpath(root, base_dir) + + for file in files: + if file.endswith('.py') and not file.startswith('__'): + # build the module path + if rel_path == '.': + # in the root directory + module_path = f"{base_package}.{file[:-3]}" + else: + # in the subdirectory + package_path = rel_path.replace(os.path.sep, '.') + module_path = f"{base_package}.{package_path}.{file[:-3]}" + + try: + importlib.import_module(module_path) + except Exception as e: + print(f"Warning: Failed to import {module_path}: {e}") + +# get the current directory and import all tools +current_dir = os.path.dirname(__file__) +import_workflows_recursively(current_dir, 'autoagent.workflows') + +# export all tool creation functions +globals().update(registry.workflows) + +__all__ = list(registry.workflows.keys()) \ No newline at end of file diff --git a/autoagent/workflows/math_solver_workflow_flow.py b/autoagent/workflows/math_solver_workflow_flow.py new file mode 100644 index 0000000..a1b08a7 --- /dev/null +++ b/autoagent/workflows/math_solver_workflow_flow.py @@ -0,0 +1,201 @@ +import asyncio +import json +import argparse +from openai import AsyncOpenAI +from openai.types.chat import ChatCompletionMessageToolCall +from autoagent.flow import default_drive, EventInput, ReturnBehavior +from autoagent.flow.dynamic import goto_events, abort_this +import re +from autoagent import MetaChain +from autoagent.types import Response +from autoagent.registry import register_workflow + +def extract_answer(response: str, key: str): + pattern = f"<{key}>(.*?)" + matches = re.findall(pattern, response) + return matches[0] if len(matches) > 0 else None + +from autoagent.agents import get_math_solver_agent + +from autoagent.agents import get_vote_aggregator_agent +@default_drive.make_event +async def on_start(event: EventInput, global_ctx): + print("start the workflow:" + 'math_solver_workflow') +@default_drive.listen_group([on_start]) +async def solve_with_gpt4(event: EventInput, global_ctx): + inputs = [{'key': 'math_problem', 'description': 'The math problem that needs to be solved.'}] + input_dict = dict() + for inp in inputs: + input_dict[inp["key"]] = global_ctx.get(inp["key"], None) + + messages = global_ctx.get('messages', []) + task = 'Solve the math problem using systematic approach and show detailed steps.' + outputs = [{'key': 'gpt4_solution', 'description': 'The solution generated by GPT-4 model.', 'condition': None, 'action': {'type': 'RESULT', 'value': None}}] + agent = get_math_solver_agent('gpt-4o-2024-08-06') + + + input_str = [] + for key, value in input_dict.items(): + input_str.append(f"The {key.replace('_', ' ')} is {value}") + input_str = "\n".join(input_str) + "\n" + query = input_str + '.\nThe task is: ' + task + '.\n' + messages.append({ + "role": "user", + "content": query + }) + client = MetaChain() + response: Response = await client.run_async(agent = agent, messages = messages, context_variables = global_ctx, debug = True) + result = response.messages[-1]["content"] + messages.extend(response.messages) + global_ctx["messages"] = messages + + for output in outputs: + ans = extract_answer(result, output["key"]) + if ans: + if output["action"]["type"] == "RESULT": + global_ctx[output["key"]] = ans + return ans + elif output["action"]["type"] == "ABORT": + return abort_this() + elif output["action"]["type"] == "GO_TO": + return goto_events([output["action"]["value"]]) + elif len(outputs) == 1: + global_ctx[output["key"]] = result + return result + raise Exception("No valid answer found") +@default_drive.listen_group([on_start]) +async def solve_with_claude(event: EventInput, global_ctx): + inputs = [{'key': 'math_problem', 'description': 'The math problem that needs to be solved.'}] + input_dict = dict() + for inp in inputs: + input_dict[inp["key"]] = global_ctx.get(inp["key"], None) + + messages = global_ctx.get('messages', []) + task = 'Solve the math problem using systematic approach and show detailed steps.' + outputs = [{'key': 'claude_solution', 'description': 'The solution generated by Claude model.', 'condition': None, 'action': {'type': 'RESULT', 'value': None}}] + agent = get_math_solver_agent('claude-3-5-sonnet-20241022') + + + input_str = [] + for key, value in input_dict.items(): + input_str.append(f"The {key.replace('_', ' ')} is {value}") + input_str = "\n".join(input_str) + "\n" + query = input_str + '.\nThe task is: ' + task + '.\n' + messages.append({ + "role": "user", + "content": query + }) + client = MetaChain() + response: Response = await client.run_async(agent = agent, messages = messages, context_variables = global_ctx, debug = True) + result = response.messages[-1]["content"] + messages.extend(response.messages) + global_ctx["messages"] = messages + + for output in outputs: + ans = extract_answer(result, output["key"]) + if ans: + if output["action"]["type"] == "RESULT": + global_ctx[output["key"]] = ans + return ans + elif output["action"]["type"] == "ABORT": + return abort_this() + elif output["action"]["type"] == "GO_TO": + return goto_events([output["action"]["value"]]) + elif len(outputs) == 1: + global_ctx[output["key"]] = result + return result + raise Exception("No valid answer found") +@default_drive.listen_group([on_start]) +async def solve_with_deepseek(event: EventInput, global_ctx): + inputs = [{'key': 'math_problem', 'description': 'The math problem that needs to be solved.'}] + input_dict = dict() + for inp in inputs: + input_dict[inp["key"]] = global_ctx.get(inp["key"], None) + + messages = global_ctx.get('messages', []) + task = 'Solve the math problem using systematic approach and show detailed steps.' + outputs = [{'key': 'deepseek_solution', 'description': 'The solution generated by Deepseek model.', 'condition': None, 'action': {'type': 'RESULT', 'value': None}}] + agent = get_math_solver_agent('deepseek/deepseek-chat') + + + input_str = [] + for key, value in input_dict.items(): + input_str.append(f"The {key.replace('_', ' ')} is {value}") + input_str = "\n".join(input_str) + "\n" + query = input_str + '.\nThe task is: ' + task + '.\n' + messages.append({ + "role": "user", + "content": query + }) + client = MetaChain() + response: Response = await client.run_async(agent = agent, messages = messages, context_variables = global_ctx, debug = True) + result = response.messages[-1]["content"] + messages.extend(response.messages) + global_ctx["messages"] = messages + + for output in outputs: + ans = extract_answer(result, output["key"]) + if ans: + if output["action"]["type"] == "RESULT": + global_ctx[output["key"]] = ans + return ans + elif output["action"]["type"] == "ABORT": + return abort_this() + elif output["action"]["type"] == "GO_TO": + return goto_events([output["action"]["value"]]) + elif len(outputs) == 1: + global_ctx[output["key"]] = result + return result + raise Exception("No valid answer found") +@default_drive.listen_group([solve_with_gpt4, solve_with_claude, solve_with_deepseek]) +async def aggregate_solutions(event: EventInput, global_ctx): + inputs = [{'key': 'gpt4_solution', 'description': 'The solution generated by GPT-4 model.'}, {'key': 'claude_solution', 'description': 'The solution generated by Claude model.'}, {'key': 'deepseek_solution', 'description': 'The solution generated by Deepseek model.'}] + input_dict = dict() + for inp in inputs: + input_dict[inp["key"]] = global_ctx.get(inp["key"], None) + + messages = global_ctx.get('messages', []) + task = 'Compare all solutions and determine the final answer through majority voting.' + outputs = [{'key': 'final_solution', 'description': 'The final agreed-upon solution after majority voting.', 'condition': None, 'action': {'type': 'RESULT', 'value': None}}] + agent = get_vote_aggregator_agent('gpt-4o-2024-08-06') + + + input_str = [] + for key, value in input_dict.items(): + input_str.append(f"The {key.replace('_', ' ')} is {value}") + input_str = "\n".join(input_str) + "\n" + query = input_str + '.\nThe task is: ' + task + '.\n' + messages.append({ + "role": "user", + "content": query + }) + client = MetaChain() + response: Response = await client.run_async(agent = agent, messages = messages, context_variables = global_ctx, debug = True) + result = response.messages[-1]["content"] + messages.extend(response.messages) + global_ctx["messages"] = messages + + for output in outputs: + ans = extract_answer(result, output["key"]) + if ans: + if output["action"]["type"] == "RESULT": + global_ctx[output["key"]] = ans + return ans + elif output["action"]["type"] == "ABORT": + return abort_this() + elif output["action"]["type"] == "GO_TO": + return goto_events([output["action"]["value"]]) + elif len(outputs) == 1: + global_ctx[output["key"]] = result + return result + raise Exception("No valid answer found") + +@register_workflow(name = 'majority_voting') +async def majority_voting(system_input: str): + storage_results = dict(math_problem = system_input) + await default_drive.invoke_event( + on_start, + global_ctx=storage_results, + ) + system_output = storage_results.get('final_solution', None) + return system_output diff --git a/constant.py b/constant.py new file mode 100644 index 0000000..ba72a2b --- /dev/null +++ b/constant.py @@ -0,0 +1,91 @@ +import os +from dotenv import load_dotenv +import platform +# utils: +load_dotenv() # 加载.env文件 +def str_to_bool(value): + """convert string to bool""" + true_values = {'true', 'yes', '1', 'on', 't', 'y'} + false_values = {'false', 'no', '0', 'off', 'f', 'n'} + + if isinstance(value, bool): + return value + + if value == None: + return None + + value = str(value).lower().strip() + if value in true_values: + return True + if value in false_values: + return False + return True # default return True + + +DOCKER_WORKPLACE_NAME = os.getenv('DOCKER_WORKPLACE_NAME', 'workplace') +GITHUB_AI_TOKEN = os.getenv('GITHUB_AI_TOKEN', None) +AI_USER = os.getenv('AI_USER', "tjb-tech") +LOCAL_ROOT = os.getenv('LOCAL_ROOT', os.getcwd()) + +DEBUG = str_to_bool(os.getenv('DEBUG', False)) + +DEFAULT_LOG = str_to_bool(os.getenv('DEFAULT_LOG', False)) +LOG_PATH = os.getenv('LOG_PATH', None) +EVAL_MODE = str_to_bool(os.getenv('EVAL_MODE', False)) +BASE_IMAGES = os.getenv('BASE_IMAGES', None) + +def get_architecture(): + machine = platform.machine().lower() + if 'x86' in machine or 'amd64' in machine or 'i386' in machine: + return "tjbtech1/metachain:amd64_latest" + elif 'arm' in machine: + return "tjbtech1/metachain:latest" + else: + return "tjbtech1/metachain:latest" +if BASE_IMAGES is None: + BASE_IMAGES = get_architecture() + +COMPLETION_MODEL = os.getenv('COMPLETION_MODEL', "claude-3-5-sonnet-20241022") +EMBEDDING_MODEL = os.getenv('EMBEDDING_MODEL', "text-embedding-3-small") + +MC_MODE = str_to_bool(os.getenv('MC_MODE', True)) + +# add Env for function call and non-function call + +FN_CALL = str_to_bool(os.getenv('FN_CALL', None)) +API_BASE_URL = os.getenv('API_BASE_URL', None) +ADD_USER = str_to_bool(os.getenv('ADD_USER', None)) + + + +NOT_SUPPORT_SENDER = ["mistral", "groq"] +MUST_ADD_USER = ["deepseek-reasoner", "o1-mini", "deepseek-r1"] + +NOT_SUPPORT_FN_CALL = ["o1-mini", "deepseek-reasoner", "deepseek-r1", "llama", "grok-2"] +NOT_USE_FN_CALL = [ "deepseek-chat"] + NOT_SUPPORT_FN_CALL + +if ADD_USER is None: + ADD_USER = False + for model in MUST_ADD_USER: + if model in COMPLETION_MODEL: + ADD_USER = True + break + +if FN_CALL is None: + FN_CALL = True + for model in NOT_USE_FN_CALL: + if model in COMPLETION_MODEL: + FN_CALL = False + break + +NON_FN_CALL = False +for model in NOT_SUPPORT_FN_CALL: + if model in COMPLETION_MODEL: + NON_FN_CALL = True + break + + +if EVAL_MODE: + DEFAULT_LOG = False + +# print(FN_CALL, NON_FN_CALL, ADD_USER) \ No newline at end of file diff --git a/docs/.gitignore b/docs/.gitignore new file mode 100644 index 0000000..b2d6de3 --- /dev/null +++ b/docs/.gitignore @@ -0,0 +1,20 @@ +# Dependencies +/node_modules + +# Production +/build + +# Generated files +.docusaurus +.cache-loader + +# Misc +.DS_Store +.env.local +.env.development.local +.env.test.local +.env.production.local + +npm-debug.log* +yarn-debug.log* +yarn-error.log* diff --git a/docs/DOC_STYLE_GUIDE.md b/docs/DOC_STYLE_GUIDE.md new file mode 100644 index 0000000..a55af79 --- /dev/null +++ b/docs/DOC_STYLE_GUIDE.md @@ -0,0 +1,48 @@ +# Documentation Style Guide + +## General Writing Principles + +- **Clarity & Conciseness**: Always prioritize clarity and brevity. Avoid unnecessary jargon or overly complex explanations. +Keep sentences short and to the point. +- **Gradual Complexity**: Start with the simplest, most basic setup, and then gradually introduce more advanced +concepts and configurations. + +## Formatting Guidelines + +### Headers + +Use **Title Case** for the first and second level headers. + +Example: + - **Basic Usage** + - **Advanced Configuration Options** + +### Lists + +When listing items or options, use bullet points to enhance readability. + +Example: + - Option A + - Option B + - Option C + +### Procedures + +For instructions or processes that need to be followed in a specific order, use numbered steps. + +Example: + 1. Step one: Do this. + 2. Step two: Complete this action. + 3. Step three: Verify the result. + +### Code Blocks + +* Use code blocks for multi-line inputs, outputs, commands and code samples. + +Example: +```bash +docker run -it \ + -e THIS=this \ + -e THAT=that + ... +``` diff --git a/docs/README.md b/docs/README.md new file mode 100644 index 0000000..0c6c2c2 --- /dev/null +++ b/docs/README.md @@ -0,0 +1,41 @@ +# Website + +This website is built using [Docusaurus](https://docusaurus.io/), a modern static website generator. + +### Installation + +``` +$ yarn +``` + +### Local Development + +``` +$ yarn start +``` + +This command starts a local development server and opens up a browser window. Most changes are reflected live without having to restart the server. + +### Build + +``` +$ yarn build +``` + +This command generates static content into the `build` directory and can be served using any static contents hosting service. + +### Deployment + +Using SSH: + +``` +$ USE_SSH=true yarn deploy +``` + +Not using SSH: + +``` +$ GIT_USER= yarn deploy +``` + +If you are using GitHub pages for hosting, this command is a convenient way to build the website and push to the `gh-pages` branch. diff --git a/docs/babel.config.js b/docs/babel.config.js new file mode 100644 index 0000000..e00595d --- /dev/null +++ b/docs/babel.config.js @@ -0,0 +1,3 @@ +module.exports = { + presets: [require.resolve('@docusaurus/core/lib/babel/preset')], +}; diff --git a/docs/docs/Dev-Guideline/dev-guide-build-your-project.md b/docs/docs/Dev-Guideline/dev-guide-build-your-project.md new file mode 100644 index 0000000..3b9830b --- /dev/null +++ b/docs/docs/Dev-Guideline/dev-guide-build-your-project.md @@ -0,0 +1,4 @@ +--- +title: Build Your Project +slug: /dev-guide-build-your-project +--- diff --git a/docs/docs/Dev-Guideline/dev-guide-create-agent.md b/docs/docs/Dev-Guideline/dev-guide-create-agent.md new file mode 100644 index 0000000..dc1b8c0 --- /dev/null +++ b/docs/docs/Dev-Guideline/dev-guide-create-agent.md @@ -0,0 +1,4 @@ +--- +title: Create Agent +slug: /dev-guide-create-agent +--- diff --git a/docs/docs/Dev-Guideline/dev-guide-create-tools.md b/docs/docs/Dev-Guideline/dev-guide-create-tools.md new file mode 100644 index 0000000..235958c --- /dev/null +++ b/docs/docs/Dev-Guideline/dev-guide-create-tools.md @@ -0,0 +1,4 @@ +--- +title: Create Tools +slug: /dev-guide-create-tools +--- diff --git a/docs/docs/Dev-Guideline/dev-guide-edit-mem.md b/docs/docs/Dev-Guideline/dev-guide-edit-mem.md new file mode 100644 index 0000000..cc00520 --- /dev/null +++ b/docs/docs/Dev-Guideline/dev-guide-edit-mem.md @@ -0,0 +1,4 @@ +--- +title: Edit Memory +slug: /dev-guide-edit-memory +--- diff --git a/docs/docs/Get-Started/get-started-installation.md b/docs/docs/Get-Started/get-started-installation.md new file mode 100644 index 0000000..05f6d0f --- /dev/null +++ b/docs/docs/Get-Started/get-started-installation.md @@ -0,0 +1,4 @@ +--- +title: Install Langflow +slug: /get-started-installation +--- diff --git a/docs/docs/Get-Started/get-started-quickstart.md b/docs/docs/Get-Started/get-started-quickstart.md new file mode 100644 index 0000000..9a0b893 --- /dev/null +++ b/docs/docs/Get-Started/get-started-quickstart.md @@ -0,0 +1,5 @@ +--- +title: Quickstart +slug: /get-started-quickstart +--- + diff --git a/docs/docs/Get-Started/welcome-to-autoagent.md b/docs/docs/Get-Started/welcome-to-autoagent.md new file mode 100644 index 0000000..c78e71f --- /dev/null +++ b/docs/docs/Get-Started/welcome-to-autoagent.md @@ -0,0 +1,29 @@ +--- +title: Welcome to AutoAgent +sidebar_position: 0 +slug: / +--- + +Welcome to AutoAgent! AutoAgent is a **Fully-Automated** and highly **Self-Developing** framework that enables users to create and deploy LLM agents through **Natural Language Alone**. + +## ✨Key Features + +* 🏆 Top Performer on the GAIA Benchmark +
AutoAgent has ranked the **#1** spot among open-sourced methods, delivering comparable performance to **OpenAI's Deep Research**. + +* 📚 Agentic-RAG with Native Self-Managing Vector Database +
AutoAgent equipped with a native self-managing vector database, outperforms industry-leading solutions like **LangChain**. + +* ✨ Agent and Workflow Create with Ease +
AutoAgent leverages natural language to effortlessly build ready-to-use **tools**, **agents** and **workflows** - no coding required. + +* 🌐 Universal LLM Support +
AutoAgent seamlessly integrates with **A Wide Range** of LLMs (e.g., OpenAI, Anthropic, Deepseek, vLLM, Grok, Huggingface ...) + +* 🔀 Flexible Interaction +
Benefit from support for both **function-calling** and **ReAct** interaction modes. + +* 🤖 Dynamic, Extensible, Lightweight +
AutoAgent is your **Personal AI Assistant**, designed to be dynamic, extensible, customized, and lightweight. + +🚀 Unlock the Future of LLM Agents. Try 🔥AutoAgent🔥 Now! \ No newline at end of file diff --git a/docs/docs/Starter-Projects/starter-projects-agentic-rag.md b/docs/docs/Starter-Projects/starter-projects-agentic-rag.md new file mode 100644 index 0000000..2a9319c --- /dev/null +++ b/docs/docs/Starter-Projects/starter-projects-agentic-rag.md @@ -0,0 +1,135 @@ +--- +title: Agentic RAG +slug: /starter-projects-agentic-rag +--- + +# Agentic RAG Implementation in AutoAgent + +Agentic RAG (Retrieval-Augmented Generation) is an intelligent retrieval system that can decide whether and how to retrieve information from a knowledge base as needed. Traditional RAG methods (such as [chunkRAG](https://github.com/chonkie-ai/chonkie), [MiniRAG](https://github.com/HKUDS/MiniRAG), [LightRAG](https://github.com/HKUDS/LightRAG), and [GraphRAG](https://github.com/microsoft/graphrag)) have limitations as they rely on predefined workflows and struggle to determine if they have acquired sufficient knowledge to answer questions. To make the RAG process more intelligent, we introduce Agentic RAG powered by [AutoAgent](https://github.com/HKUDS/AutoAgent), implementing intelligent storage, retrieval, and response. + +## System Architecture + +### 1. Required Imports +```python +from constant import DOCKER_WORKPLACE_NAME +from autoagent.environment.docker_container import init_container +from autoagent.io_utils import read_yaml_file, get_md5_hash_bytext +from autoagent.agents import get_rag_agent +from autoagent.core import AutoAgent +from autoagent.environment.docker_env import DockerEnv, DockerConfig, with_env +import argparse +import asyncio +import csv +from tqdm import trange +import os +import json +import time +``` + +### 2. Environment Configuration +```python +def get_env(container_name: str = 'gaia_test', + model: str = 'gpt-4o-mini-2024-07-18', + git_clone: bool = False, + setup_package: str = 'lite_pkgs'): + workplace_name = DOCKER_WORKPLACE_NAME + docker_config = DockerConfig( + container_name=container_name, + workplace_name=workplace_name, + communication_port=12345, + conda_path='/home/user/micromamba' + ) + docker_env = DockerEnv(docker_config) + return docker_env +``` + +The system runs in a Docker container, providing an isolated environment with the following main configurations: +- Container name +- Working directory +- Communication port +- Conda environment path + +### 3. RAG Agent Setup +```python +async def main(container_name: str = 'gaia_test', model: str = 'gpt-4o-mini-2024-07-18', git_clone: bool = False, setup_package: str = 'lite_pkgs', test_pull_name: str = 'test_pull_1010', debug: bool = True, task_instructions: str = None): + workplace_name = DOCKER_WORKPLACE_NAME + # Docker environment is optional + # docker_env = get_env(container_name, model, git_clone, setup_package, test_pull_name, debug) + # docker_env.init_container() + + task_instructions = "YOUR TASK" + + rag_agent = get_rag_agent(model)#, rag_env=docker_env) + mc = AutoAgent() +``` + +The system uses the AutoAgent framework to manage RAG agents, with key features including: +- Asynchronous operation support +- Configurable language models +- Flexible message handling mechanism + +### 4. Query Processing Flow +```python +context_variables = { + "working_dir": DOCKER_WORKPLACE_NAME, + "user_query": task_instructions +} +messages = [{"role": "user", "content": task_instructions}] +response = await mc.run_async( + agent=codeact_agent, + messages=messages, + max_turns=10, + context_variables=context_variables, + debug=debug +) +``` + +Query processing includes the following steps: +1. Setting context variables +2. Building message format +3. Asynchronous agent execution +4. Controlling maximum conversation turns +5. Debug mode support + +## Usage + +We put a basic usage example in [`AutoAgent/evaluation/multihoprag`](https://github.com/HKUDS/AutoAgent/tree/main/evaluation/multihoprag). + + +### 1. Basic Usage +```bash +current_dir=$(dirname "$(readlink -f "$0")") + +cd $current_dir +cd ../ +export DOCKER_WORKPLACE_NAME=workplace_rag +export EVAL_MODE=True +export DEBUG=True +export BASE_IMAGES=tjbtech1/gaia-bookworm:v2 +export COMPLETION_MODEL=claude-3-5-sonnet-20241022 + +python run_rag.py --model gpt-4o-mini-2024-07-18 --container_name gaia_test +``` + +### 2. Parameter Description +- `--container_name`: Docker container name +- `--model`: Language model to use +- `--git_clone`: Whether to clone code +- `--setup_package`: Package type to install +- `--debug`: Whether to enable debug mode + +## Key Features + +1. **Asynchronous Processing**: Using `asyncio` for improved processing efficiency +2. **Containerized Deployment**: Using Docker for environment consistency +3. **Flexible Configuration**: Support for various models and parameter configurations +4. **Batch Processing**: Support for batch query processing +5. **Result Tracking**: Saving queries and responses for evaluation and analysis + +## Important Notes + +1. Ensure proper Docker environment configuration +2. Check model access permissions and configurations +3. Set appropriate maximum conversation turns +4. Maintain data format consistency +5. Regular backup of result files \ No newline at end of file diff --git a/docs/docs/Starter-Projects/starter-projects-auto-deep-research.md b/docs/docs/Starter-Projects/starter-projects-auto-deep-research.md new file mode 100644 index 0000000..8db0e1b --- /dev/null +++ b/docs/docs/Starter-Projects/starter-projects-auto-deep-research.md @@ -0,0 +1,5 @@ +--- +title: Auto Deep Research +slug: /starter-projects-auto-deep-research +--- + diff --git a/docs/docs/Starter-Projects/starter-projects-nl-to-agent.md b/docs/docs/Starter-Projects/starter-projects-nl-to-agent.md new file mode 100644 index 0000000..21e86c8 --- /dev/null +++ b/docs/docs/Starter-Projects/starter-projects-nl-to-agent.md @@ -0,0 +1,4 @@ +--- +title: From Natural Language to Agent +slug: /starter-projects-nl-to-agent +--- diff --git a/docs/docs/User-Guideline/user-guide-daily-tasks.md b/docs/docs/User-Guideline/user-guide-daily-tasks.md new file mode 100644 index 0000000..9f329f5 --- /dev/null +++ b/docs/docs/User-Guideline/user-guide-daily-tasks.md @@ -0,0 +1,5 @@ +--- +title: For Daily Tasks +slug: /user-guide-daily-tasks +--- + diff --git a/docs/docs/User-Guideline/user-guide-how-to-create-agent.md b/docs/docs/User-Guideline/user-guide-how-to-create-agent.md new file mode 100644 index 0000000..591c296 --- /dev/null +++ b/docs/docs/User-Guideline/user-guide-how-to-create-agent.md @@ -0,0 +1,5 @@ +--- +title: How to create an agent with Natural Language +slug: /user-guide-how-to-create-agent +--- + diff --git a/docs/docs/python/python.md b/docs/docs/python/python.md new file mode 100644 index 0000000..fde7708 --- /dev/null +++ b/docs/docs/python/python.md @@ -0,0 +1,3 @@ +# Python Docs + +Docs will appear here after deployment. diff --git a/docs/docs/python/sidebar.json b/docs/docs/python/sidebar.json new file mode 100644 index 0000000..d9035ad --- /dev/null +++ b/docs/docs/python/sidebar.json @@ -0,0 +1,5 @@ +{ + "items": ["python/python"], + "label": "Backend", + "type": "category" +} diff --git a/docs/docusaurus.config.ts b/docs/docusaurus.config.ts new file mode 100644 index 0000000..00c6c31 --- /dev/null +++ b/docs/docusaurus.config.ts @@ -0,0 +1,78 @@ +import type * as Preset from "@docusaurus/preset-classic"; +import type { Config } from "@docusaurus/types"; +import { themes as prismThemes } from "prism-react-renderer"; + +const config: Config = { + title: "AutoAgent", + tagline: "Fully-Automated & Zero-Code LLM Agent Framework", + favicon: "img/metachain_logo.svg", + + // Set the production url of your site here + url: "https://autoagent-ai.github.io", + baseUrl: "/", + + // GitHub pages deployment config. + organizationName: "autoagent-ai", + projectName: "autoagent-ai.github.io", + trailingSlash: false, + deploymentBranch: "main", + + onBrokenLinks: "throw", + onBrokenMarkdownLinks: "warn", + + markdown: { + mermaid: true, + }, + themes: ['@docusaurus/theme-mermaid'], + presets: [ + [ + "classic", + { + docs: { + path: "docs", + routeBasePath: "docs", + sidebarPath: "./sidebars.ts", + exclude: [ + "**/*.test.{js,jsx,ts,tsx}", + "**/__tests__/**", + ], + }, + blog: { + showReadingTime: true, + }, + theme: { + customCss: "./src/css/custom.css", + }, + } satisfies Preset.Options, + ], + ], + themeConfig: { + image: "img/docusaurus.png", + navbar: { + title: "AutoAgent", + logo: { + alt: "AutoAgent", + src: "img/metachain_logo.svg", + }, + items: [ + { + type: "docSidebar", + sidebarId: "docsSidebar", + position: "left", + label: "Docs", + }, + { + href: "https://github.com/HKUDS/AutoAgent", + label: "GitHub", + position: "right", + }, + ], + }, + prism: { + theme: prismThemes.oneLight, + darkTheme: prismThemes.oneDark, + }, + } satisfies Preset.ThemeConfig, +}; + +export default config; diff --git a/docs/i18n/fr/code.json b/docs/i18n/fr/code.json new file mode 100644 index 0000000..685d93b --- /dev/null +++ b/docs/i18n/fr/code.json @@ -0,0 +1,406 @@ +{ + "footer.title": { + "message": "OpenHands" + }, + "footer.docs": { + "message": "Documents" + }, + "footer.community": { + "message": "Communauté" + }, + "footer.copyright": { + "message": "© {year} OpenHands" + }, + "faq.title": { + "message": "Questions Fréquemment Posées", + "description": "FAQ Title" + }, + "faq.description": { + "message": "Questions Fréquemment Posées" + }, + "faq.section.title.1": { + "message": "Qu'est-ce qu'OpenHands ?", + "description": "First Section Title" + }, + "faq.section.highlight": { + "message": "OpenHands", + "description": "Highlight Text" + }, + "faq.section.description.1": { + "message": "est un ingénieur logiciel autonome qui peut résoudre des tâches d'ingénierie logicielle et de navigation web à tout moment. Il peut exécuter des requêtes en sciences des données, telles que \"Trouver le nombre de demandes de pull à l'repository OpenHands dans les derniers mois\", et des tâches d'ingénierie logicielle, comme \"Veuillez ajouter des tests à ce fichier et vérifier si tous les tests passent. Si ce n'est pas le cas, réparez le fichier.\"", + "description": "Description for OpenHands" + }, + "faq.section.description.2": { + "message": "De plus, OpenHands est une plateforme et communauté pour les développeurs d'agents qui souhaitent tester et évaluer de nouveaux agents.", + "description": "Further Description for OpenHands" + }, + "faq.section.title.2": { + "message": "Support", + "description": "Support Section Title" + }, + "faq.section.support.answer": { + "message": "Si vous rencontrez un problème que d'autres utilisateurs peuvent également avoir, merci de le signaler sur {githubLink}. Si vous avez des difficultés à l'installation ou des questions générales, rejoignez-vous sur {discordLink} ou {slackLink}.", + "description": "Support Answer" + }, + "faq.section.title.3": { + "message": "Comment résoudre un problème sur GitHub avec OpenHands ?", + "description": "GitHub Issue Section Title" + }, + "faq.section.github.steps.intro": { + "message": "Pour résoudre un problème sur GitHub en utilisant OpenHands, envoyez une commande à OpenHands demandant qu'il suit des étapes comme les suivantes :", + "description": "GitHub Steps Introduction" + }, + "faq.section.github.step1": { + "message": "Lisez l'issue https://github.com/All-Hands-AI/OpenHands/issues/1611", + "description": "GitHub Step 1" + }, + "faq.section.github.step2": { + "message": "Cloner le dépôt et vérifier une nouvelle branche", + "description": "GitHub Step 2" + }, + "faq.section.github.step3": { + "message": "Sur la base des instructions dans la description de l'issue, modifiez les fichiers pour résoudre le problème", + "description": "GitHub Step 3" + }, + "faq.section.github.step4": { + "message": "Pousser le résultat à GitHub en utilisant la variable d'environnement GITHUB_TOKEN", + "description": "GitHub Step 4" + }, + "faq.section.github.step5": { + "message": "Dites-moi le lien que je dois utiliser pour envoyer une demande de pull", + "description": "GitHub Step 5" + }, + "faq.section.github.steps.preRun": { + "message": "Avant de lancer OpenHands, vous pouvez faire :", + "description": "GitHub Steps Pre-Run" + }, + "faq.section.github.steps.tokenInfo": { + "message": "où XXX est un jeton GitHub que vous avez créé et qui a les autorisations pour pousser dans le dépôt OpenHands. Si vous n'avez pas d'autorisations de modification du dépôt OpenHands, vous devrez peut-être changer cela en :", + "description": "GitHub Steps Token Info" + }, + "faq.section.github.steps.usernameInfo": { + "message": "où USERNAME est votre nom GitHub.", + "description": "GitHub Steps Username Info" + }, + "faq.section.title.4": { + "message": "Comment OpenHands est-il différent de Devin ?", + "description": "Devin Section Title" + }, + "faq.section.openhands.linkText": { + "message": "Devin", + "description": "Devin Link Text" + }, + "faq.section.openhands.description": { + "message": "est un produit commercial par Cognition Inc., qui a servi d'inspiration initiale pour OpenHands. Les deux visent à bien faire le travail d'ingénierie logicielle, mais vous pouvez télécharger, utiliser et modifier OpenHands, tandis que Devin peut être utilisé uniquement via le site de Cognition. De plus, OpenHands a évolué au-delà de l'inspiration initiale, et est maintenant un écosystème communautaire pour le développement d'agents en général, et nous serions ravis de vous voir rejoindre et", + "description": "Devin Description" + }, + "faq.section.openhands.contribute": { + "message": "contribuer", + "description": "Contribute Link" + }, + "faq.section.title.5": { + "message": "Comment OpenHands est-il différent de ChatGPT ?", + "description": "ChatGPT Section Title" + }, + "faq.section.chatgpt.description": { + "message": "ChatGPT vous pouvez accéder en ligne, il ne se connecte pas aux fichiers locaux et ses capacités d'exécution du code sont limitées. Alors qu'il peut écrire du code, mais c'est difficile à tester ou à exécuter.", + "description": "ChatGPT Description" + }, + "homepage.description": { + "message": "Génération d'code AI pour l'ingénierie logicielle.", + "description": "The homepage description" + }, + "homepage.getStarted": { + "message": "Commencer" + }, + "welcome.message": { + "message": "Bienvenue à OpenHands, un système d'IA autonome ingénieur logiciel capable d'exécuter des tâches d'ingénierie complexes et de collaborer activement avec les utilisateurs sur les projets de développement logiciel." + }, + "theme.ErrorPageContent.title": { + "message": "Cette page a planté.", + "description": "The title of the fallback page when the page crashed" + }, + "theme.BackToTopButton.buttonAriaLabel": { + "message": "Retourner en haut de la page", + "description": "The ARIA label for the back to top button" + }, + "theme.blog.archive.title": { + "message": "Archives", + "description": "The page & hero title of the blog archive page" + }, + "theme.blog.archive.description": { + "message": "Archives", + "description": "The page & hero description of the blog archive page" + }, + "theme.blog.paginator.navAriaLabel": { + "message": "Pagination des listes d'articles du blog", + "description": "The ARIA label for the blog pagination" + }, + "theme.blog.paginator.newerEntries": { + "message": "Nouvelles entrées", + "description": "The label used to navigate to the newer blog posts page (previous page)" + }, + "theme.blog.paginator.olderEntries": { + "message": "Anciennes entrées", + "description": "The label used to navigate to the older blog posts page (next page)" + }, + "theme.blog.post.paginator.navAriaLabel": { + "message": "Pagination des articles du blog", + "description": "The ARIA label for the blog posts pagination" + }, + "theme.blog.post.paginator.newerPost": { + "message": "Article plus récent", + "description": "The blog post button label to navigate to the newer/previous post" + }, + "theme.blog.post.paginator.olderPost": { + "message": "Article plus ancien", + "description": "The blog post button label to navigate to the older/next post" + }, + "theme.blog.post.plurals": { + "message": "Un article|{count} articles", + "description": "Pluralized label for \"{count} posts\". Use as much plural forms (separated by \"|\") as your language support (see https://www.unicode.org/cldr/cldr-aux/charts/34/supplemental/language_plural_rules.html)" + }, + "theme.blog.tagTitle": { + "message": "{nPosts} tags avec « {tagName} »", + "description": "The title of the page for a blog tag" + }, + "theme.tags.tagsPageLink": { + "message": "Voir tous les tags", + "description": "The label of the link targeting the tag list page" + }, + "theme.colorToggle.ariaLabel": { + "message": "Basculer entre le mode sombre et clair (actuellement {mode})", + "description": "The ARIA label for the navbar color mode toggle" + }, + "theme.colorToggle.ariaLabel.mode.dark": { + "message": "mode sombre", + "description": "The name for the dark color mode" + }, + "theme.colorToggle.ariaLabel.mode.light": { + "message": "mode clair", + "description": "The name for the light color mode" + }, + "theme.docs.breadcrumbs.navAriaLabel": { + "message": "Bouton de navigation des liens de la page", + "description": "The ARIA label for the breadcrumbs" + }, + "theme.docs.DocCard.categoryDescription.plurals": { + "message": "1 élément|{count} éléments", + "description": "The default description for a category card in the generated index about how many items this category includes" + }, + "theme.docs.paginator.navAriaLabel": { + "message": "Pages de documentation", + "description": "The ARIA label for the docs pagination" + }, + "theme.docs.paginator.previous": { + "message": "Précédent", + "description": "The label used to navigate to the previous doc" + }, + "theme.docs.paginator.next": { + "message": "Suivant", + "description": "The label used to navigate to the next doc" + }, + "theme.docs.tagDocListPageTitle.nDocsTagged": { + "message": "Un document tagué|{count} documents tagués", + "description": "Pluralized label for \"{count} docs tagged\". Use as much plural forms (separated by \"|\") as your language support (see https://www.unicode.org/cldr/cldr-aux/charts/34/supplemental/language_plural_rules.html)" + }, + "theme.docs.tagDocListPageTitle": { + "message": "{nDocsTagged} avec \"{tagName}\"", + "description": "The title of the page for a docs tag" + }, + "theme.docs.versionBadge.label": { + "message": "Version: {versionLabel}" + }, + "theme.docs.versions.unreleasedVersionLabel": { + "message": "Ceci est la documentation de la prochaine version {versionLabel} de {siteTitle}.", + "description": "The label used to tell the user that he's browsing an unreleased doc version" + }, + "theme.docs.versions.unmaintainedVersionLabel": { + "message": "Ceci est la documentation de {siteTitle} {versionLabel}, qui n'est plus activement maintenue.", + "description": "The label used to tell the user that he's browsing an unmaintained doc version" + }, + "theme.docs.versions.latestVersionSuggestionLabel": { + "message": "Pour une documentation à jour, consultez la {latestVersionLink} ({versionLabel}).", + "description": "The label used to tell the user to check the latest version" + }, + "theme.docs.versions.latestVersionLinkLabel": { + "message": "dernière version", + "description": "The label used for the latest version suggestion link label" + }, + "theme.common.editThisPage": { + "message": "Éditer cette page", + "description": "The link label to edit the current page" + }, + "theme.common.headingLinkTitle": { + "message": "Lien direct vers {heading}", + "description": "Title for link to heading" + }, + "theme.lastUpdated.atDate": { + "message": " le {date}", + "description": "The words used to describe on which date a page has been last updated" + }, + "theme.lastUpdated.byUser": { + "message": " par {user}", + "description": "The words used to describe by who the page has been last updated" + }, + "theme.lastUpdated.lastUpdatedAtBy": { + "message": "Dernière mise à jour{atDate}{byUser}", + "description": "The sentence used to display when a page has been last updated, and by who" + }, + "theme.navbar.mobileVersionsDropdown.label": { + "message": "Versions", + "description": "The label for the navbar versions dropdown on mobile view" + }, + "theme.NotFound.title": { + "message": "Page introuvable", + "description": "The title of the 404 page" + }, + "theme.tags.tagsListLabel": { + "message": "Tags :", + "description": "The label alongside a tag list" + }, + "theme.admonition.caution": { + "message": "prudence", + "description": "The default label used for the Caution admonition (:::caution)" + }, + "theme.admonition.danger": { + "message": "danger", + "description": "The default label used for the Danger admonition (:::danger)" + }, + "theme.admonition.info": { + "message": "information", + "description": "The default label used for the Info admonition (:::info)" + }, + "theme.admonition.note": { + "message": "remarque", + "description": "The default label used for the Note admonition (:::note)" + }, + "theme.admonition.tip": { + "message": "astuce", + "description": "The default label used for the Tip admonition (:::tip)" + }, + "theme.admonition.warning": { + "message": "prudence", + "description": "The default label used for the Warning admonition (:::warning)" + }, + "theme.AnnouncementBar.closeButtonAriaLabel": { + "message": "Fermer", + "description": "The ARIA label for close button of announcement bar" + }, + "theme.blog.sidebar.navAriaLabel": { + "message": "Navigation vers les articles récents du blog", + "description": "The ARIA label for recent posts in the blog sidebar" + }, + "theme.CodeBlock.copied": { + "message": "Copié", + "description": "The copied button label on code blocks" + }, + "theme.CodeBlock.copyButtonAriaLabel": { + "message": "Copier le code", + "description": "The ARIA label for copy code blocks button" + }, + "theme.CodeBlock.copy": { + "message": "Copier", + "description": "The copy button label on code blocks" + }, + "theme.CodeBlock.wordWrapToggle": { + "message": "Activer/désactiver le retour à la ligne", + "description": "The title attribute for toggle word wrapping button of code block lines" + }, + "theme.DocSidebarItem.expandCategoryAriaLabel": { + "message": "Développer la catégorie '{label}' de la barre latérale", + "description": "The ARIA label to expand the sidebar category" + }, + "theme.DocSidebarItem.collapseCategoryAriaLabel": { + "message": "Réduire la catégorie '{label}' de la barre latérale", + "description": "The ARIA label to collapse the sidebar category" + }, + "theme.NavBar.navAriaLabel": { + "message": "Main", + "description": "The ARIA label for the main navigation" + }, + "theme.navbar.mobileLanguageDropdown.label": { + "message": "Langues", + "description": "The label for the mobile language switcher dropdown" + }, + "theme.NotFound.p1": { + "message": "Nous n'avons pas trouvé ce que vous recherchez.", + "description": "The first paragraph of the 404 page" + }, + "theme.NotFound.p2": { + "message": "Veuillez contacter le propriétaire du site qui vous a lié à l'URL d'origine et leur faire savoir que leur lien est cassé.", + "description": "The 2nd paragraph of the 404 page" + }, + "theme.TOCCollapsible.toggleButtonLabel": { + "message": "Sur cette page", + "description": "The label used by the button on the collapsible TOC component" + }, + "theme.blog.post.readMore": { + "message": "Lire plus", + "description": "The label used in blog post item excerpts to link to full blog posts" + }, + "theme.blog.post.readMoreLabel": { + "message": "En savoir plus sur {title}", + "description": "The ARIA label for the link to full blog posts from excerpts" + }, + "theme.blog.post.readingTime.plurals": { + "message": "Une minute de lecture|{readingTime} minutes de lecture", + "description": "Pluralized label for \"{readingTime} min read\". Use as much plural forms (separated by \"|\") as your language support (see https://www.unicode.org/cldr/cldr-aux/charts/34/supplemental/language_plural_rules.html)" + }, + "theme.docs.breadcrumbs.home": { + "message": "Page d'accueil", + "description": "The ARIA label for the home page in the breadcrumbs" + }, + "theme.docs.sidebar.collapseButtonTitle": { + "message": "Réduire le menu latéral", + "description": "The title attribute for collapse button of doc sidebar" + }, + "theme.docs.sidebar.collapseButtonAriaLabel": { + "message": "Réduire le menu latérale", + "description": "The title attribute for collapse button of doc sidebar" + }, + "theme.docs.sidebar.navAriaLabel": { + "message": "Barre de navigation latérale des docs", + "description": "The ARIA label for the sidebar navigation" + }, + "theme.docs.sidebar.closeSidebarButtonAriaLabel": { + "message": "Fermer la barre de navigation", + "description": "The ARIA label for close button of mobile sidebar" + }, + "theme.navbar.mobileSidebarSecondaryMenu.backButtonLabel": { + "message": "← Retour au menu principal", + "description": "The label of the back button to return to main menu, inside the mobile navbar sidebar secondary menu (notably used to display the docs sidebar)" + }, + "theme.docs.sidebar.toggleSidebarButtonAriaLabel": { + "message": "Ouvrir/fermer la barre de navigation", + "description": "The ARIA label for hamburger menu button of mobile navigation" + }, + "theme.docs.sidebar.expandButtonTitle": { + "message": "Déplier le menu latéral", + "description": "The ARIA label and title attribute for expand button of doc sidebar" + }, + "theme.docs.sidebar.expandButtonAriaLabel": { + "message": "Déployer le menu latérale", + "description": "The ARIA label and title attribute for expand button of doc sidebar" + }, + "theme.ErrorPageContent.tryAgain": { + "message": "Réessayer", + "description": "The label of the button to try again rendering when the React error boundary captures an error" + }, + "theme.common.skipToMainContent": { + "message": "Aller directement au contenu principal", + "description": "The skip to content label used for accessibility, allowing to rapidly navigate to main content with keyboard tab/enter navigation" + }, + "theme.tags.tagsPageTitle": { + "message": "Tags", + "description": "The title of the tag list page" + }, + "theme.unlistedContent.title": { + "message": "Page non répertoriée", + "description": "The unlisted content banner title" + }, + "theme.unlistedContent.message": { + "message": "Cette page n'est pas répertoriée. Les moteurs de recherche ne l'indexeront pas, et seuls les utilisateurs ayant un lien direct peuvent y accéder.", + "description": "The unlisted content banner message" + } +} diff --git a/docs/i18n/fr/docusaurus-plugin-content-blog/options.json b/docs/i18n/fr/docusaurus-plugin-content-blog/options.json new file mode 100644 index 0000000..0eee9a2 --- /dev/null +++ b/docs/i18n/fr/docusaurus-plugin-content-blog/options.json @@ -0,0 +1,14 @@ +{ + "title": { + "message": "Blog", + "description": "The title for the blog used in SEO" + }, + "description": { + "message": "Blog", + "description": "The description for the blog used in SEO" + }, + "sidebar.title": { + "message": "Articles récents", + "description": "The label for the left sidebar" + } +} diff --git a/docs/i18n/fr/docusaurus-plugin-content-docs/current.json b/docs/i18n/fr/docusaurus-plugin-content-docs/current.json new file mode 100644 index 0000000..eeeb461 --- /dev/null +++ b/docs/i18n/fr/docusaurus-plugin-content-docs/current.json @@ -0,0 +1,18 @@ +{ + "version.label": { + "message": "Next", + "description": "The label for version current" + }, + "sidebar.docsSidebar.category.🤖 Backends LLM": { + "message": "🤖 Backends LLM", + "description": "The label for category 🤖 Backends LLM in sidebar docsSidebar" + }, + "sidebar.docsSidebar.category.🚧 Dépannage": { + "message": "🚧 Dépannage", + "description": "The label for category 🚧 Dépannage in sidebar docsSidebar" + }, + "sidebar.apiSidebar.category.Backend": { + "message": "Backend", + "description": "The label for category Backend in sidebar apiSidebar" + } +} diff --git a/docs/i18n/fr/docusaurus-plugin-content-docs/current/python/python.md b/docs/i18n/fr/docusaurus-plugin-content-docs/current/python/python.md new file mode 100644 index 0000000..7233096 --- /dev/null +++ b/docs/i18n/fr/docusaurus-plugin-content-docs/current/python/python.md @@ -0,0 +1,5 @@ + + +# Documentation Python + +La documentation apparaîtra ici après le déploiement. diff --git a/docs/i18n/fr/docusaurus-plugin-content-docs/current/python/sidebar.json b/docs/i18n/fr/docusaurus-plugin-content-docs/current/python/sidebar.json new file mode 100644 index 0000000..53257ae --- /dev/null +++ b/docs/i18n/fr/docusaurus-plugin-content-docs/current/python/sidebar.json @@ -0,0 +1,5 @@ +{ + "items": ["python/python"], + "label": "Backend", + "type": "categorie" +} diff --git a/docs/i18n/fr/docusaurus-plugin-content-docs/current/usage/about.md b/docs/i18n/fr/docusaurus-plugin-content-docs/current/usage/about.md new file mode 100644 index 0000000..a386795 --- /dev/null +++ b/docs/i18n/fr/docusaurus-plugin-content-docs/current/usage/about.md @@ -0,0 +1,28 @@ + + +# À propos d'OpenHands + +## Stratégie de recherche + +La réplication complète d'applications de niveau production avec des LLM est une entreprise complexe. Notre stratégie implique : + +1. **Recherche technique fondamentale :** Se concentrer sur la recherche fondamentale pour comprendre et améliorer les aspects techniques de la génération et de la gestion du code +2. **Capacités spécialisées :** Améliorer l'efficacité des composants de base grâce à la curation des données, aux méthodes d'entraînement, et plus encore +3. **Planification des tâches :** Développer des capacités pour la détection des bugs, la gestion des bases de code et l'optimisation +4. **Évaluation :** Établir des métriques d'évaluation complètes pour mieux comprendre et améliorer nos modèles + +## Agent par défaut + +Notre Agent par défaut est actuellement le [CodeActAgent](agents), qui est capable de générer du code et de gérer des fichiers. + +## Construit avec + +OpenHands est construit en utilisant une combinaison de frameworks et de bibliothèques puissants, fournissant une base solide pour son développement. Voici les principales technologies utilisées dans le projet : + +![FastAPI](https://img.shields.io/badge/FastAPI-black?style=for-the-badge) ![uvicorn](https://img.shields.io/badge/uvicorn-black?style=for-the-badge) ![LiteLLM](https://img.shields.io/badge/LiteLLM-black?style=for-the-badge) ![Docker](https://img.shields.io/badge/Docker-black?style=for-the-badge) ![Ruff](https://img.shields.io/badge/Ruff-black?style=for-the-badge) ![MyPy](https://img.shields.io/badge/MyPy-black?style=for-the-badge) ![LlamaIndex](https://img.shields.io/badge/LlamaIndex-black?style=for-the-badge) ![React](https://img.shields.io/badge/React-black?style=for-the-badge) + +Veuillez noter que la sélection de ces technologies est en cours et que des technologies supplémentaires peuvent être ajoutées ou des technologies existantes peuvent être supprimées à mesure que le projet évolue. Nous nous efforçons d'adopter les outils les plus appropriés et les plus efficaces pour améliorer les capacités d'OpenHands. + +## Licence + +Distribué sous la [Licence](https://github.com/All-Hands-AI/OpenHands/blob/main/LICENSE) MIT. diff --git a/docs/i18n/fr/docusaurus-plugin-content-docs/current/usage/agents.md b/docs/i18n/fr/docusaurus-plugin-content-docs/current/usage/agents.md new file mode 100644 index 0000000..d283f42 --- /dev/null +++ b/docs/i18n/fr/docusaurus-plugin-content-docs/current/usage/agents.md @@ -0,0 +1,25 @@ + + +# 🧠 Agent Principal et Capacités + +## CodeActAgent + +### Description + +Cet agent implémente l'idée de CodeAct ([article](https://arxiv.org/abs/2402.01030), [tweet](https://twitter.com/xingyaow_/status/1754556835703751087)) qui consolide les **act**ions des agents LLM dans un espace d'action de **code** unifié à la fois pour la _simplicité_ et la _performance_. + +L'idée conceptuelle est illustrée ci-dessous. À chaque tour, l'agent peut : + +1. **Converser** : Communiquer avec les humains en langage naturel pour demander des clarifications, des confirmations, etc. +2. **CodeAct** : Choisir d'effectuer la tâche en exécutant du code + +- Exécuter n'importe quelle commande Linux `bash` valide +- Exécuter n'importe quel code `Python` valide avec [un interpréteur Python interactif](https://ipython.org/). Ceci est simulé via une commande `bash`, voir le système de plugin ci-dessous pour plus de détails. + +![image](https://github.com/All-Hands-AI/OpenHands/assets/38853559/92b622e3-72ad-4a61-8f41-8c040b6d5fb3) + +### Démo + +https://github.com/All-Hands-AI/OpenHands/assets/38853559/f592a192-e86c-4f48-ad31-d69282d5f6ac + +_Exemple de CodeActAgent avec `gpt-4-turbo-2024-04-09` effectuant une tâche de science des données (régression linéaire)_. diff --git a/docs/i18n/fr/docusaurus-plugin-content-docs/current/usage/architecture.mdx b/docs/i18n/fr/docusaurus-plugin-content-docs/current/usage/architecture.mdx new file mode 100644 index 0000000..fa6436a --- /dev/null +++ b/docs/i18n/fr/docusaurus-plugin-content-docs/current/usage/architecture.mdx @@ -0,0 +1,50 @@ +--- +sidebar_position: 4 +--- + +# 🏛️ Aperçu de l'Architecture Système + +Voici un aperçu de haut niveau de l'architecture du système. Le système est divisé en deux composants principaux : le frontend et le backend. Le frontend est responsable de la gestion des interactions avec l'utilisateur et de l'affichage des résultats. Le backend est responsable de la gestion de la logique métier et de l'exécution des agents. + +![system_architecture.svg](/img/system_architecture.svg) + +Cet aperçu est simplifié pour montrer les principaux composants et leurs interactions. Pour une vue plus détaillée de l'architecture du backend, consultez la section [Architecture du Backend](#backend-architecture-fr). + +# Architecture du Backend {#backend-architecture-fr} + +_**Avertissement**: L'architecture du backend est en cours de développement et est sujette à modifications. Le schéma suivant montre l'architecture actuelle du backend basée sur le commit indiqué dans le pied de page du schéma._ + +![backend_architecture.svg](/img/backend_architecture.svg) + +
+ Mise à jour de ce Schéma +
+ La génération du schéma d'architecture du backend est partiellement automatisée. + Le schéma est généré à partir des annotations de type dans le code en utilisant l'outil py2puml. + Le schéma est ensuite revu manuellement, ajusté et exporté en PNG et SVG. + + ## Prérequis + + - Un environnement Python dans lequel openhands est exécutable + (selon les instructions du fichier README.md à la racine du dépôt) + - [py2puml](https://github.com/lucsorel/py2puml) installé + +## Étapes + +1. Générez automatiquement le schéma en exécutant la commande suivante depuis la racine du dépôt : + `py2puml openhands openhands > docs/architecture/backend_architecture.puml` + +2. Ouvrez le fichier généré dans un éditeur PlantUML, par exemple Visual Studio Code avec l'extension PlantUML ou [PlantText](https://www.planttext.com/) + +3. Révisez le PUML généré et apportez toutes les modifications nécessaires au schéma (ajoutez les parties manquantes, corrigez les erreurs, améliorez l'agencement). + _py2puml crée le schéma à partir des annotations de type dans le code, donc les annotations de type manquantes ou incorrectes peuvent entraîner un schéma incomplet ou incorrect._ + +4. Examinez la différence entre le nouveau schéma et le précédent et vérifiez manuellement si les modifications sont correctes. + _Assurez-vous de ne pas supprimer les parties ajoutées manuellement au schéma par le passé et qui sont toujours pertinentes._ + +5. Ajoutez le hash du commit qui a été utilisé pour générer le schéma dans le pied de page du schéma. + +6. Exporte le schéma sous forme de fichiers PNG et SVG et remplacez les schémas existants dans le répertoire `docs/architecture`. Cela peut être fait avec (par exemple [PlantText](https://www.planttext.com/)) + +
+
diff --git a/docs/i18n/fr/docusaurus-plugin-content-docs/current/usage/architecture/backend.mdx b/docs/i18n/fr/docusaurus-plugin-content-docs/current/usage/architecture/backend.mdx new file mode 100644 index 0000000..d422c59 --- /dev/null +++ b/docs/i18n/fr/docusaurus-plugin-content-docs/current/usage/architecture/backend.mdx @@ -0,0 +1,54 @@ + + +# 🏛️ Architecture du Système + +
+ OpenHands System Architecture Diagram Jul 4 2024 +

Diagramme de l'Architecture du Système OpenHands (4 juillet 2024)

+
+ +Ceci est une vue d'ensemble de haut niveau de l'architecture du système. Le système est divisé en deux composants principaux : le frontend et le backend. Le frontend est responsable de la gestion des interactions utilisateur et de l'affichage des résultats. Le backend est responsable de la gestion de la logique métier et de l'exécution des agents. + +# Architecture du Frontend {#frontend-architecture-fr} + +![system_architecture.svg](/img/system_architecture.svg) + +Cette vue d'ensemble est simplifiée pour montrer les principaux composants et leurs interactions. Pour une vue plus détaillée de l'architecture du backend, voir la section Architecture du Backend ci-dessous. + +# Architecture du Backend {#backend-architecture-fr} + +_**Avertissement** : L'architecture du backend est en cours de développement et est sujette à changement. Le diagramme suivant montre l'architecture actuelle du backend basée sur le commit indiqué dans le pied de page du diagramme._ + +![backend_architecture.svg](/img/backend_architecture.svg) + +
+ Mise à jour de ce Diagramme +
+ La génération du diagramme d'architecture du backend est partiellement automatisée. + Le diagramme est généré à partir des indications de type dans le code en utilisant l'outil py2puml. Le diagramme est ensuite manuellement revu, ajusté et exporté en PNG et SVG. + + ## Prérequis + + - Environnement python fonctionnel dans lequel openhands est exécutable + (selon les instructions du fichier README.md à la racine du dépôt) + - [py2puml](https://github.com/lucsorel/py2puml) installé + +## Étapes + +1. Générer automatiquement le diagramme en exécutant la commande suivante depuis la racine du dépôt : + `py2puml openhands openhands > docs/architecture/backend_architecture.puml` + +2. Ouvrir le fichier généré dans un éditeur PlantUML, par ex. Visual Studio Code avec l'extension PlantUML ou [PlantText](https://www.planttext.com/) + +3. Revoir le PUML généré et effectuer tous les ajustements nécessaires au diagramme (ajouter les parties manquantes, corriger les erreurs, améliorer le positionnement). + _py2puml crée le diagramme en se basant sur les indications de type dans le code, donc des indications manquantes ou incorrectes peuvent entraîner un diagramme incomplet ou incorrect._ + +4. Revoir la différence entre le nouveau diagramme et le précédent et vérifier manuellement si les changements sont corrects. + _S'assurer de ne pas supprimer des parties qui ont été ajoutées manuellement au diagramme par le passé et qui sont toujours pertinentes._ + +5. Ajouter le hash du commit qui a été utilisé pour générer le diagramme dans le pied de page du diagramme. + +6. Exporter le diagramme sous forme de fichiers PNG et SVG et remplacer les diagrammes existants dans le répertoire `docs/architecture`. Cela peut être fait avec (par ex. [PlantText](https://www.planttext.com/)) + +
+
diff --git a/docs/i18n/fr/docusaurus-plugin-content-docs/current/usage/architecture/runtime.md b/docs/i18n/fr/docusaurus-plugin-content-docs/current/usage/architecture/runtime.md new file mode 100644 index 0000000..71e121d --- /dev/null +++ b/docs/i18n/fr/docusaurus-plugin-content-docs/current/usage/architecture/runtime.md @@ -0,0 +1,138 @@ + + +# 📦 Runtime Docker + +Le Runtime Docker d'OpenHands est le composant principal qui permet l'exécution sécurisée et flexible des actions des agents d'IA. +Il crée un environnement en bac à sable (sandbox) en utilisant Docker, où du code arbitraire peut être exécuté en toute sécurité sans risquer le système hôte. + +## Pourquoi avons-nous besoin d'un runtime en bac à sable ? + +OpenHands doit exécuter du code arbitraire dans un environnement sécurisé et isolé pour plusieurs raisons : + +1. Sécurité : L'exécution de code non fiable peut poser des risques importants pour le système hôte. Un environnement en bac à sable empêche le code malveillant d'accéder ou de modifier les ressources du système hôte +2. Cohérence : Un environnement en bac à sable garantit que l'exécution du code est cohérente sur différentes machines et configurations, éliminant les problèmes du type "ça fonctionne sur ma machine" +3. Contrôle des ressources : Le bac à sable permet un meilleur contrôle de l'allocation et de l'utilisation des ressources, empêchant les processus incontrôlés d'affecter le système hôte +4. Isolation : Différents projets ou utilisateurs peuvent travailler dans des environnements isolés sans interférer les uns avec les autres ou avec le système hôte +5. Reproductibilité : Les environnements en bac à sable facilitent la reproduction des bugs et des problèmes, car l'environnement d'exécution est cohérent et contrôlable + +## Comment fonctionne le Runtime ? + +Le système Runtime d'OpenHands utilise une architecture client-serveur implémentée avec des conteneurs Docker. Voici un aperçu de son fonctionnement : + +```mermaid +graph TD + A[Image Docker personnalisée fournie par l'utilisateur] --> B[Backend OpenHands] + B -->|Construit| C[Image OH Runtime] + C -->|Lance| D[Exécuteur d'actions] + D -->|Initialise| E[Navigateur] + D -->|Initialise| F[Shell Bash] + D -->|Initialise| G[Plugins] + G -->|Initialise| L[Serveur Jupyter] + + B -->|Génère| H[Agent] + B -->|Génère| I[EventStream] + I <--->|Exécute l'action pour + obtenir l'observation + via l'API REST + | D + + H -->|Génère l'action| I + I -->|Obtient l'observation| H + + subgraph "Conteneur Docker" + D + E + F + G + L + end +``` + +1. Entrée utilisateur : L'utilisateur fournit une image Docker de base personnalisée +2. Construction de l'image : OpenHands construit une nouvelle image Docker (l'"image OH runtime") basée sur l'image fournie par l'utilisateur. Cette nouvelle image inclut le code spécifique à OpenHands, principalement le "client runtime" +3. Lancement du conteneur : Lorsqu'OpenHands démarre, il lance un conteneur Docker en utilisant l'image OH runtime +4. Initialisation du serveur d'exécution des actions : Le serveur d'exécution des actions initialise un `ActionExecutor` à l'intérieur du conteneur, mettant en place les composants nécessaires comme un shell bash et chargeant les plugins spécifiés +5. Communication : Le backend OpenHands (`openhands/runtime/impl/eventstream/eventstream_runtime.py`) communique avec le serveur d'exécution des actions via une API RESTful, envoyant des actions et recevant des observations +6. Exécution des actions : Le client runtime reçoit les actions du backend, les exécute dans l'environnement en bac à sable et renvoie les observations +7. Retour des observations : Le serveur d'exécution des actions renvoie les résultats d'exécution au backend OpenHands sous forme d'observations + + +Le rôle du client : +- Il agit comme un intermédiaire entre le backend OpenHands et l'environnement en bac à sable +- Il exécute différents types d'actions (commandes shell, opérations sur les fichiers, code Python, etc.) en toute sécurité dans le conteneur +- Il gère l'état de l'environnement en bac à sable, y compris le répertoire de travail courant et les plugins chargés +- Il formate et renvoie les observations au backend, assurant une interface cohérente pour le traitement des résultats + + +## Comment OpenHands construit et maintient les images OH Runtime + +L'approche d'OpenHands pour la construction et la gestion des images runtime assure l'efficacité, la cohérence et la flexibilité dans la création et la maintenance des images Docker pour les environnements de production et de développement. + +Consultez le [code pertinent](https://github.com/All-Hands-AI/OpenHands/blob/main/openhands/runtime/utils/runtime_build.py) si vous souhaitez plus de détails. + +### Système de balises d'images + +OpenHands utilise un système à trois balises pour ses images runtime afin d'équilibrer la reproductibilité et la flexibilité. +Les balises peuvent être dans l'un des 2 formats suivants : + +- **Balise versionnée** : `oh_v{openhands_version}_{base_image}` (ex : `oh_v0.9.9_nikolaik_s_python-nodejs_t_python3.12-nodejs22`) +- **Balise de verrouillage** : `oh_v{openhands_version}_{16_digit_lock_hash}` (ex : `oh_v0.9.9_1234567890abcdef`) +- **Balise source** : `oh_v{openhands_version}_{16_digit_lock_hash}_{16_digit_source_hash}` + (ex : `oh_v0.9.9_1234567890abcdef_1234567890abcdef`) + + +#### Balise source - La plus spécifique + +Il s'agit des 16 premiers chiffres du MD5 du hash du répertoire pour le répertoire source. Cela donne un hash +uniquement pour la source d'openhands + + +#### Balise de verrouillage + +Ce hash est construit à partir des 16 premiers chiffres du MD5 de : +- Le nom de l'image de base sur laquelle l'image a été construite (ex : `nikolaik/python-nodejs:python3.12-nodejs22`) +- Le contenu du `pyproject.toml` inclus dans l'image. +- Le contenu du `poetry.lock` inclus dans l'image. + +Cela donne effectivement un hash pour les dépendances d'Openhands indépendamment du code source. + +#### Balise versionnée - La plus générique + +Cette balise est une concaténation de la version d'openhands et du nom de l'image de base (transformé pour s'adapter au standard des balises). + +#### Processus de construction + +Lors de la génération d'une image... + +- **Pas de reconstruction** : OpenHands vérifie d'abord si une image avec la même **balise source la plus spécifique** existe. S'il existe une telle image, + aucune construction n'est effectuée - l'image existante est utilisée. +- **Reconstruction la plus rapide** : OpenHands vérifie ensuite si une image avec la **balise de verrouillage générique** existe. S'il existe une telle image, + OpenHands construit une nouvelle image basée sur celle-ci, en contournant toutes les étapes d'installation (comme `poetry install` et + `apt-get`) sauf une opération finale pour copier le code source actuel. La nouvelle image est balisée avec une + balise **source** uniquement. +- **Reconstruction correcte** : Si ni une balise **source** ni une balise **de verrouillage** n'existe, une image sera construite sur la base de l'image avec la balise **versionnée**. + Dans l'image avec la balise versionnée, la plupart des dépendances devraient déjà être installées, ce qui permet de gagner du temps. +- **Reconstruction la plus lente** : Si les trois balises n'existent pas, une toute nouvelle image est construite à partir de + l'image de base (ce qui est une opération plus lente). Cette nouvelle image est balisée avec toutes les balises **source**, **de verrouillage** et **versionnée**. + +Cette approche de balisage permet à OpenHands de gérer efficacement les environnements de développement et de production. + +1. Un code source et un Dockerfile identiques produisent toujours la même image (via des balises basées sur des hashs) +2. Le système peut reconstruire rapidement les images lorsque des changements mineurs se produisent (en s'appuyant sur des images compatibles récentes) +3. La balise **de verrouillage** (ex : `runtime:oh_v0.9.3_1234567890abcdef`) pointe toujours vers la dernière version pour une combinaison particulière d'image de base, de dépendances et de version d'OpenHands + +## Système de plugins du Runtime + +Le Runtime d'OpenHands prend en charge un système de plugins qui permet d'étendre les fonctionnalités et de personnaliser l'environnement d'exécution. Les plugins sont initialisés lorsque le client runtime démarre. + +Consultez [un exemple de plugin Jupyter ici](https://github.com/All-Hands-AI/OpenHands/blob/ecf4aed28b0cf7c18d4d8ff554883ba182fc6bdd/openhands/runtime/plugins/jupyter/__init__.py#L21-L55) si vous souhaitez implémenter votre propre plugin. + +*Plus de détails sur le système de plugins sont encore en construction - les contributions sont les bienvenues !* + +Aspects clés du système de plugins : + +1. Définition des plugins : Les plugins sont définis comme des classes Python qui héritent d'une classe de base `Plugin` +2. Enregistrement des plugins : Les plugins disponibles sont enregistrés dans un dictionnaire `ALL_PLUGINS` +3. Spécification des plugins : Les plugins sont associés à `Agent.sandbox_plugins: list[PluginRequirement]`. Les utilisateurs peuvent spécifier quels plugins charger lors de l'initialisation du runtime +4. Initialisation : Les plugins sont initialisés de manière asynchrone lorsque le client runtime démarre +5. Utilisation : Le client runtime peut utiliser les plugins initialisés pour étendre ses capacités (par exemple, le JupyterPlugin pour exécuter des cellules IPython) diff --git a/docs/i18n/fr/docusaurus-plugin-content-docs/current/usage/configuration-options.md b/docs/i18n/fr/docusaurus-plugin-content-docs/current/usage/configuration-options.md new file mode 100644 index 0000000..7115c85 --- /dev/null +++ b/docs/i18n/fr/docusaurus-plugin-content-docs/current/usage/configuration-options.md @@ -0,0 +1,395 @@ +# Options de configuration + +Ce guide détaille toutes les options de configuration disponibles pour OpenHands, vous aidant à personnaliser son comportement et à l'intégrer avec d'autres services. + +:::note +Si vous exécutez en [Mode GUI](https://docs.all-hands.dev/modules/usage/how-to/gui-mode), les paramètres disponibles dans l'interface utilisateur des paramètres auront toujours +la priorité. +::: + +--- + +# Table des matières + +1. [Configuration de base](#configuration-de-base) + - [Clés API](#clés-api) + - [Espace de travail](#espace-de-travail) + - [Débogage et journalisation](#débogage-et-journalisation) + - [Gestion des sessions](#gestion-des-sessions) + - [Trajectoires](#trajectoires) + - [Stockage de fichiers](#stockage-de-fichiers) + - [Gestion des tâches](#gestion-des-tâches) + - [Configuration du bac à sable](#configuration-du-bac-à-sable) + - [Divers](#divers) +2. [Configuration LLM](#configuration-llm) + - [Informations d'identification AWS](#informations-didentification-aws) + - [Configuration de l'API](#configuration-de-lapi) + - [Fournisseur LLM personnalisé](#fournisseur-llm-personnalisé) + - [Embeddings](#embeddings) + - [Gestion des messages](#gestion-des-messages) + - [Sélection du modèle](#sélection-du-modèle) + - [Nouvelles tentatives](#nouvelles-tentatives) + - [Options avancées](#options-avancées) +3. [Configuration de l'agent](#configuration-de-lagent) + - [Configuration du micro-agent](#configuration-du-micro-agent) + - [Configuration de la mémoire](#configuration-de-la-mémoire) + - [Configuration LLM](#configuration-llm-2) + - [Configuration de l'espace d'action](#configuration-de-lespace-daction) + - [Utilisation du micro-agent](#utilisation-du-micro-agent) +4. [Configuration du bac à sable](#configuration-du-bac-à-sable-2) + - [Exécution](#exécution) + - [Image de conteneur](#image-de-conteneur) + - [Mise en réseau](#mise-en-réseau) + - [Linting et plugins](#linting-et-plugins) + - [Dépendances et environnement](#dépendances-et-environnement) + - [Évaluation](#évaluation) +5. [Configuration de sécurité](#configuration-de-sécurité) + - [Mode de confirmation](#mode-de-confirmation) + - [Analyseur de sécurité](#analyseur-de-sécurité) + +--- + +## Configuration de base + +Les options de configuration de base sont définies dans la section `[core]` du fichier `config.toml`. + +**Clés API** +- `e2b_api_key` + - Type : `str` + - Valeur par défaut : `""` + - Description : Clé API pour E2B + +- `modal_api_token_id` + - Type : `str` + - Valeur par défaut : `""` + - Description : ID du jeton API pour Modal + +- `modal_api_token_secret` + - Type : `str` + - Valeur par défaut : `""` + - Description : Secret du jeton API pour Modal + +**Espace de travail** +- `workspace_base` + - Type : `str` + - Valeur par défaut : `"./workspace"` + - Description : Chemin de base pour l'espace de travail + +- `cache_dir` + - Type : `str` + - Valeur par défaut : `"/tmp/cache"` + - Description : Chemin du répertoire de cache + +**Débogage et journalisation** +- `debug` + - Type : `bool` + - Valeur par défaut : `false` + - Description : Activer le débogage + +- `disable_color` + - Type : `bool` + - Valeur par défaut : `false` + - Description : Désactiver la couleur dans la sortie du terminal + +**Trajectoires** +- `save_trajectory_path` + - Type : `str` + - Valeur par défaut : `"./trajectories"` + - Description : Chemin pour stocker les trajectoires (peut être un dossier ou un fichier). Si c'est un dossier, les trajectoires seront enregistrées dans un fichier nommé avec l'ID de session et l'extension .json, dans ce dossier. + +**Stockage de fichiers** +- `file_store_path` + - Type : `str` + - Valeur par défaut : `"/tmp/file_store"` + - Description : Chemin de stockage des fichiers + +- `file_store` + - Type : `str` + - Valeur par défaut : `"memory"` + - Description : Type de stockage de fichiers + +- `file_uploads_allowed_extensions` + - Type : `list of str` + - Valeur par défaut : `[".*"]` + - Description : Liste des extensions de fichiers autorisées pour les téléchargements + +- `file_uploads_max_file_size_mb` + - Type : `int` + - Valeur par défaut : `0` + - Description : Taille maximale des fichiers pour les téléchargements, en mégaoctets + +- `file_uploads_restrict_file_types` + - Type : `bool` + - Valeur par défaut : `false` + - Description : Restreindre les types de fichiers pour les téléchargements de fichiers + +- `file_uploads_allowed_extensions` + - Type : `list of str` + - Valeur par défaut : `[".*"]` + - Description : Liste des extensions de fichiers autorisées pour les téléchargements + +**Gestion des tâches** +- `max_budget_per_task` + - Type : `float` + - Valeur par défaut : `0.0` + - Description : Budget maximal par tâche (0.0 signifie aucune limite) + +- `max_iterations` + - Type : `int` + - Valeur par défaut : `100` + - Description : Nombre maximal d'itérations + +**Configuration du bac à sable** +- `workspace_mount_path_in_sandbox` + - Type : `str` + - Valeur par défaut : `"/workspace"` + - Description : Chemin de montage de l'espace de travail dans le bac à sable + +- `workspace_mount_path` + - Type : `str` + - Valeur par défaut : `""` + - Description : Chemin de montage de l'espace de travail + +- `workspace_mount_rewrite` + - Type : `str` + - Valeur par défaut : `""` + - Description : Chemin pour réécrire le chemin de montage de l'espace de travail. Vous pouvez généralement ignorer cela, cela fait référence à des cas spéciaux d'exécution à l'intérieur d'un autre conteneur. + +**Divers** +- `run_as_openhands` + - Type : `bool` + - Valeur par défaut : `true` + - Description : Exécuter en tant qu'OpenHands + +- `runtime` + - Type : `str` + - Valeur par défaut : `"docker"` + - Description : Environnement d'exécution + +- `default_agent` + - Type : `str` + - Valeur par défaut : `"CodeActAgent"` + - Description : Nom de l'agent par défaut + +- `jwt_secret` + - Type : `str` + - Valeur par défaut : `uuid.uuid4().hex` + - Description : Secret JWT pour l'authentification. Veuillez le définir sur votre propre valeur. + +## Configuration LLM + +Les options de configuration LLM (Large Language Model) sont définies dans la section `[llm]` du fichier `config.toml`. + +Pour les utiliser avec la commande docker, passez `-e LLM_