mirrors/patchy631--ai-engineering-hub
1
0
Fork 0
Code Issues Pull Requests Actions Deployments Models Agent Notes Packages Projects Releases Wiki Activity

chore: import upstream snapshot with attribution

Browse Source
This commit is contained in:
wehub-resource-sync
2026-07-13 12:37:47 +08:00
commit 7653f56fed
1422 changed files with 359026 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
documentation-writer-flow/.gitignore
+21
View File
@@ -0,0 +1,21 @@
__pycache__/
.venv/
.uv/
.uv_cache/
.env
.env.*
.pyenv/
*.pyc
*.py[cod]
*.so
*.egg
*.egg-info/
dist/
build/
.ipynb_checkpoints/
*.log
*.pid
*.out
*.err
.DS_Store
Thumbs.db
documentation-writer-flow/.python-version
+1
View File
@@ -0,0 +1 @@
3.12
documentation-writer-flow/Building_Code_Documentation_Agents_Local.ipynb
+988
View File
@@ -0,0 +1,988 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Building Code Documentation Agents with CrewAI"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Prerequisites "
]
},
{
"cell_type": "code",
"execution_count": 19,
"metadata": {},
"outputs": [],
"source": [
"from crewai import LLM\n",
"\n",
"def load_llm():\n",
" llm = LLM(\n",
" # model=\"ollama/deepseek-r1:7b\",\n",
" model=\"ollama/llama3.2\",\n",
" base_url=\"http://localhost:11434\"\n",
" )\n",
" return llm"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Initialization and Setup\n",
"Initial imports for the CrewAI Flow and Crew and setting up the environment"
]
},
{
"cell_type": "code",
"execution_count": 20,
"metadata": {},
"outputs": [],
"source": [
"# Importing necessary libraries\n",
"import yaml\n",
"import subprocess\n",
"from pathlib import Path\n",
"from pydantic import BaseModel\n",
"\n",
"# Importing Crew related components\n",
"from crewai import Agent, Task, Crew\n",
"\n",
"# Importing CrewAI Flow related components\n",
"from crewai.flow.flow import Flow, listen, start\n",
"\n",
"# Apply a patch to allow nested asyncio loops in Jupyter\n",
"import nest_asyncio\n",
"nest_asyncio.apply()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Define the project URL\n",
"\n",
"In this demo, a sample repository is provided for you. However, feel free to test this on other public repositories! "
]
},
{
"cell_type": "code",
"execution_count": 21,
"metadata": {},
"outputs": [],
"source": [
"project_url = \"https://github.com/crewAIInc/nvidia-demo\""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Plan for our Flow\n",
"\n",
"1. Clone the repository for the project\n",
"2. Plan the documentation for the project **[Planning Crew]** \n",
"3. Create the documentation for the project **[Documentation Crew]**"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Create Pydantic Schema\n",
"\n",
"Initial strucutre data we will use to capture the output of the planning crew"
]
},
{
"cell_type": "code",
"execution_count": 22,
"metadata": {},
"outputs": [],
"source": [
"# Define data structures to capture documentation planning output\n",
"class DocItem(BaseModel):\n",
" \"\"\"Represents a documentation item\"\"\"\n",
" title: str\n",
" description: str\n",
" prerequisites: str\n",
" examples: list[str]\n",
" goal: str\n",
"\n",
"class DocPlan(BaseModel):\n",
" \"\"\"Documentation plan\"\"\"\n",
" overview: str\n",
" docs: list[DocItem]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Optimizing for Llama 3.2 Prompting Template\n",
"\n",
"When using different models the ability to go a lower level and change the prompting template can drastically improve the performance of the model, you want to make sure to watch for the model's training prompt patterns and adjust accordingly.\n",
"\n",
"For Meta's Llama you can find it [in here](https://www.llama.com/docs/model-cards-and-prompt-formats/llama3_1/#prompt-template)"
]
},
{
"cell_type": "code",
"execution_count": 23,
"metadata": {},
"outputs": [],
"source": [
"# Agents Prompting Template for Llama 3.3\n",
"system_template=\"\"\"<|begin_of_text|><|start_header_id|>system<|end_header_id|>{{ .System }}<|eot_id|>\"\"\"\n",
"prompt_template=\"\"\"<|start_header_id|>user<|end_header_id|>{{ .Prompt }}<|eot_id|>\"\"\"\n",
"response_template=\"\"\"<|start_header_id|>assistant<|end_header_id|>{{ .Response }}<|eot_id|>\"\"\""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Create Planning Crew\n",
"\n",
"Crew of AI Agents to strategize and create a documentation plan."
]
},
{
"cell_type": "code",
"execution_count": 24,
"metadata": {},
"outputs": [],
"source": [
"from crewai_tools import (\n",
" DirectoryReadTool,\n",
" FileReadTool,\n",
")\n",
"\n",
"# Load agent and task configurations from YAML files\n",
"with open('config/planner_agents.yaml', 'r') as f:\n",
" agents_config = yaml.safe_load(f)\n",
"\n",
"with open('config/planner_tasks.yaml', 'r') as f:\n",
" tasks_config = yaml.safe_load(f)\n",
"\n",
"code_explorer = Agent(\n",
" config=agents_config['code_explorer'],\n",
" system_template=system_template,\n",
" prompt_template=prompt_template,\n",
" response_template=response_template,\n",
" tools=[\n",
" DirectoryReadTool(),\n",
" FileReadTool()\n",
" ],\n",
" llm=load_llm()\n",
")\n",
"documentation_planner = Agent(\n",
" config=agents_config['documentation_planner'],\n",
" system_template=system_template,\n",
" prompt_template=prompt_template,\n",
" response_template=response_template,\n",
" tools=[\n",
" DirectoryReadTool(),\n",
" FileReadTool()\n",
" ],\n",
" llm=load_llm()\n",
")\n",
"\n",
"analyze_codebase = Task(\n",
" config=tasks_config['analyze_codebase'],\n",
" agent=code_explorer\n",
")\n",
"create_documentation_plan = Task(\n",
" config=tasks_config['create_documentation_plan'],\n",
" agent=documentation_planner,\n",
" output_pydantic=DocPlan\n",
")\n",
"\n",
"planning_crew = Crew(\n",
" agents=[code_explorer, documentation_planner],\n",
" tasks=[analyze_codebase, create_documentation_plan],\n",
" verbose=False\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Create Documentation Crew\n",
"\n",
"Crew of AI Agents to execute the documentation plan and create the documentation.\n",
"Creating a guardrail to check the mermaid syntax in the documentation."
]
},
{
"cell_type": "code",
"execution_count": 25,
"metadata": {},
"outputs": [],
"source": [
"from crewai.tasks import TaskOutput\n",
"import re\n",
"\n",
"def check_mermaid_syntax(task_output: TaskOutput):\n",
" text = task_output.raw\n",
"\n",
" # Find all mermaid code blocks in the text\n",
" mermaid_blocks = re.findall(r'```mermaid\\n(.*?)\\n```', text, re.DOTALL)\n",
"\n",
" for block in mermaid_blocks:\n",
" diagram_text = block.strip()\n",
" lines = diagram_text.split('\\n')\n",
" corrected_lines = []\n",
"\n",
" for line in lines:\n",
" corrected_line = re.sub(r'\\|.*?\\|>', lambda match: match.group(0).replace('|>', '|'), line)\n",
" corrected_lines.append(corrected_line)\n",
"\n",
" text = text.replace(block, \"\\n\".join(corrected_lines))\n",
"\n",
" task_output.raw = text\n",
" return (True, task_output)"
]
},
{
"cell_type": "code",
"execution_count": 26,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"/Users/akshay/Eigen/ai-engineering-hub/documentation-writer-flow/.venv/lib/python3.12/site-packages/ollama/_types.py:81: PydanticDeprecatedSince211: Accessing the 'model_fields' attribute on the instance is deprecated. Instead, you should access this attribute from the model class. Deprecated in Pydantic V2.11 to be removed in V3.0.\n",
" if key in self.model_fields:\n",
"/Users/akshay/Eigen/ai-engineering-hub/documentation-writer-flow/.venv/lib/python3.12/site-packages/embedchain/embedder/ollama.py:27: LangChainDeprecationWarning: The class `OllamaEmbeddings` was deprecated in LangChain 0.3.1 and will be removed in 1.0.0. An updated version of the class exists in the :class:`~langchain-ollama package and should be used instead. To use it run `pip install -U :class:`~langchain-ollama` and import as `from :class:`~langchain_ollama import OllamaEmbeddings``.\n",
" embeddings = OllamaEmbeddings(model=self.config.model, base_url=config.base_url)\n"
]
}
],
"source": [
"from crewai_tools import (\n",
" DirectoryReadTool,\n",
" FileReadTool,\n",
" WebsiteSearchTool\n",
")\n",
"\n",
"# Load agent and task configurations from YAML files\n",
"with open('config/documentation_agents.yaml', 'r') as f:\n",
" agents_config = yaml.safe_load(f)\n",
"\n",
"with open('config/documentation_tasks.yaml', 'r') as f:\n",
" tasks_config = yaml.safe_load(f)\n",
"\n",
"overview_writer = Agent(config=agents_config['overview_writer'], tools=[\n",
" DirectoryReadTool(),\n",
" FileReadTool(),\n",
" WebsiteSearchTool(\n",
" website=\"https://mermaid.js.org/intro/\",\n",
" config=dict(\n",
" embedder=dict(\n",
" provider=\"ollama\",\n",
" config=dict(\n",
" model=\"nomic-embed-text\",\n",
" ),\n",
" )\n",
" )\n",
" )\n",
" ],\n",
" llm=load_llm()\n",
")\n",
"\n",
"documentation_reviewer = Agent(config=agents_config['documentation_reviewer'], tools=[\n",
" DirectoryReadTool(directory=\"docs/\", name=\"Check existing documentation folder\"),\n",
" FileReadTool(),\n",
" ],\n",
" llm=load_llm()\n",
")\n",
"\n",
"draft_documentation = Task(\n",
" config=tasks_config['draft_documentation'],\n",
" agent=overview_writer\n",
")\n",
"\n",
"qa_review_documentation = Task(\n",
" config=tasks_config['qa_review_documentation'],\n",
" agent=documentation_reviewer,\n",
" guardrail=check_mermaid_syntax,\n",
" max_retries=5\n",
")\n",
"\n",
"documentation_crew = Crew(\n",
" agents=[overview_writer, documentation_reviewer],\n",
" tasks=[draft_documentation, qa_review_documentation],\n",
" verbose=False\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Create Documentation Flow\n",
"\n",
"A Flow to create the documentation for the project where we will use the planning crew to plan the documentation and the documentation crew to create the documentation"
]
},
{
"cell_type": "code",
"execution_count": 27,
"metadata": {},
"outputs": [],
"source": [
"\n",
"from typing import List\n",
"\n",
"\n",
"class DocumentationState(BaseModel):\n",
" \"\"\"\n",
" State for the documentation flow\n",
" \"\"\"\n",
" project_url: str = project_url\n",
" repo_path: Path = \"workdir/\"\n",
" docs: List[str] = []\n",
"\n",
"class CreateDocumentationFlow(Flow[DocumentationState]):\n",
" # Clone the repository, initial step\n",
" # No need for AI Agents on this step, so we just use regular Python code\n",
" @start()\n",
" def clone_repo(self):\n",
" print(f\"# Cloning repository: {self.state.project_url}\\n\")\n",
" # Extract repo name from URL\n",
" repo_name = self.state.project_url.split(\"/\")[-1]\n",
" self.state.repo_path = f\"{self.state.repo_path}{repo_name}\"\n",
"\n",
" # Check if directory exists\n",
" if Path(self.state.repo_path).exists():\n",
" print(f\"# Repository directory already exists at {self.state.repo_path}\\n\")\n",
" subprocess.run([\"rm\", \"-rf\", self.state.repo_path])\n",
" print(\"# Removed existing directory\\n\")\n",
"\n",
" # Clone the repository\n",
" subprocess.run([\"git\", \"clone\", self.state.project_url, self.state.repo_path])\n",
" return self.state\n",
"\n",
" @listen(clone_repo)\n",
" def plan_docs(self):\n",
" print(f\"# Planning documentation for: {self.state.repo_path}\\n\")\n",
" result = planning_crew.kickoff(inputs={'repo_path': self.state.repo_path})\n",
" print(f\"# Planned docs for {self.state.repo_path}:\")\n",
" for doc in result.pydantic.docs:\n",
" print(f\" - {doc.title}\")\n",
" return result\n",
"\n",
" @listen(plan_docs)\n",
" def save_plan(self, plan):\n",
" with open(\"docs/plan.json\", \"w\") as f:\n",
" f.write(plan.raw)\n",
"\n",
" @listen(plan_docs)\n",
" def create_docs(self, plan):\n",
" for doc in plan.pydantic.docs:\n",
" print(f\"\\n# Creating documentation for: {doc.title}\")\n",
" result = documentation_crew.kickoff(inputs={\n",
" 'repo_path': self.state.repo_path,\n",
" 'title': doc.title,\n",
" 'overview': plan.pydantic.overview,\n",
" 'description': doc.description,\n",
" 'prerequisites': doc.prerequisites,\n",
" 'examples': '\\n'.join(doc.examples),\n",
" 'goal': doc.goal\n",
" })\n",
"\n",
" # Save documentation to file in docs folder\n",
" docs_dir = Path(\"docs\")\n",
" docs_dir.mkdir(exist_ok=True)\n",
" title = doc.title.lower().replace(\" \", \"_\") + \".mdx\"\n",
" self.state.docs.append(str(docs_dir / title))\n",
" with open(docs_dir / title, \"w\") as f:\n",
" f.write(result.raw)\n",
" print(f\"\\n# Documentation created for: {self.state.repo_path}\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Run Documentation Flow\n",
"\n",
"After running this cell, check the `docs` directory for the generated documentation. "
]
},
{
"cell_type": "code",
"execution_count": 29,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #000080; text-decoration-color: #000080\">╭──────────────────────────────────────────────── Flow Execution ─────────────────────────────────────────────────╮</span>\n",
"<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
"<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">Starting Flow Execution</span> <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
"<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0\">Name: </span><span style=\"color: #000080; text-decoration-color: #000080\">CreateDocumentationFlow</span> <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
"<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0\">ID: </span><span style=\"color: #000080; text-decoration-color: #000080\">7e73bcbc-5fd6-4742-97f7-2f28264632af</span> <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
"<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
"<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
"<span style=\"color: #000080; text-decoration-color: #000080\">╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</span>\n",
"</pre>\n"
],
"text/plain": [
"\u001b[34m╭─\u001b[0m\u001b[34m───────────────────────────────────────────────\u001b[0m\u001b[34m Flow Execution \u001b[0m\u001b[34m────────────────────────────────────────────────\u001b[0m\u001b[34m─╮\u001b[0m\n",
"\u001b[34m│\u001b[0m \u001b[34m│\u001b[0m\n",
"\u001b[34m│\u001b[0m \u001b[1;34mStarting Flow Execution\u001b[0m \u001b[34m│\u001b[0m\n",
"\u001b[34m│\u001b[0m \u001b[37mName: \u001b[0m\u001b[34mCreateDocumentationFlow\u001b[0m \u001b[34m│\u001b[0m\n",
"\u001b[34m│\u001b[0m \u001b[37mID: \u001b[0m\u001b[34m7e73bcbc-5fd6-4742-97f7-2f28264632af\u001b[0m \u001b[34m│\u001b[0m\n",
"\u001b[34m│\u001b[0m \u001b[34m│\u001b[0m\n",
"\u001b[34m│\u001b[0m \u001b[34m│\u001b[0m\n",
"\u001b[34m╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯\u001b[0m\n"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">\n",
"</pre>\n"
],
"text/plain": [
"\n"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">🌊 Flow: </span><span style=\"color: #000080; text-decoration-color: #000080\">CreateDocumentationFlow</span>\n",
"<span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0\"> ID: </span><span style=\"color: #000080; text-decoration-color: #000080\">7e73bcbc-5fd6-4742-97f7-2f28264632af</span>\n",
"└── <span style=\"color: #808000; text-decoration-color: #808000\">🧠 Starting Flow...</span>\n",
"</pre>\n"
],
"text/plain": [
"\u001b[1;34m🌊 Flow: \u001b[0m\u001b[34mCreateDocumentationFlow\u001b[0m\n",
"\u001b[37m ID: \u001b[0m\u001b[34m7e73bcbc-5fd6-4742-97f7-2f28264632af\u001b[0m\n",
"└── \u001b[33m🧠 Starting Flow...\u001b[0m\n"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">\n",
"</pre>\n"
],
"text/plain": [
"\n"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\u001b[1m\u001b[35m Flow started with ID: 7e73bcbc-5fd6-4742-97f7-2f28264632af\u001b[00m\n"
]
},
{
"data": {
"text/html": [
"<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">🌊 Flow: </span><span style=\"color: #000080; text-decoration-color: #000080\">CreateDocumentationFlow</span>\n",
"<span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0\"> ID: </span><span style=\"color: #000080; text-decoration-color: #000080\">7e73bcbc-5fd6-4742-97f7-2f28264632af</span>\n",
"├── <span style=\"color: #808000; text-decoration-color: #808000\">🧠 Starting Flow...</span>\n",
"└── <span style=\"color: #808000; text-decoration-color: #808000; font-weight: bold\">🔄 Running: clone_repo</span>\n",
"</pre>\n"
],
"text/plain": [
"\u001b[1;34m🌊 Flow: \u001b[0m\u001b[34mCreateDocumentationFlow\u001b[0m\n",
"\u001b[37m ID: \u001b[0m\u001b[34m7e73bcbc-5fd6-4742-97f7-2f28264632af\u001b[0m\n",
"├── \u001b[33m🧠 Starting Flow...\u001b[0m\n",
"└── \u001b[1;33m🔄 Running:\u001b[0m\u001b[1;33m clone_repo\u001b[0m\n"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">\n",
"</pre>\n"
],
"text/plain": [
"\n"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"# Cloning repository: https://github.com/crewAIInc/nvidia-demo\n",
"\n",
"# Repository directory already exists at workdir/nvidia-demo\n",
"\n",
"# Removed existing directory\n",
"\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"Cloning into 'workdir/nvidia-demo'...\n"
]
},
{
"data": {
"text/html": [
"<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">🌊 Flow: </span><span style=\"color: #000080; text-decoration-color: #000080\">CreateDocumentationFlow</span>\n",
"<span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0\"> ID: </span><span style=\"color: #000080; text-decoration-color: #000080\">7e73bcbc-5fd6-4742-97f7-2f28264632af</span>\n",
"├── <span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0\">Flow Method Step</span>\n",
"└── <span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\">✅ Completed: clone_repo</span>\n",
"</pre>\n"
],
"text/plain": [
"\u001b[1;34m🌊 Flow: \u001b[0m\u001b[34mCreateDocumentationFlow\u001b[0m\n",
"\u001b[37m ID: \u001b[0m\u001b[34m7e73bcbc-5fd6-4742-97f7-2f28264632af\u001b[0m\n",
"├── \u001b[37mFlow Method Step\u001b[0m\n",
"└── \u001b[1;32m✅ Completed:\u001b[0m\u001b[1;32m clone_repo\u001b[0m\n"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">\n",
"</pre>\n"
],
"text/plain": [
"\n"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">🌊 Flow: </span><span style=\"color: #000080; text-decoration-color: #000080\">CreateDocumentationFlow</span>\n",
"<span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0\"> ID: </span><span style=\"color: #000080; text-decoration-color: #000080\">7e73bcbc-5fd6-4742-97f7-2f28264632af</span>\n",
"├── <span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0\">Flow Method Step</span>\n",
"├── <span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\">✅ Completed: clone_repo</span>\n",
"└── <span style=\"color: #808000; text-decoration-color: #808000; font-weight: bold\">🔄 Running: plan_docs</span>\n",
"</pre>\n"
],
"text/plain": [
"\u001b[1;34m🌊 Flow: \u001b[0m\u001b[34mCreateDocumentationFlow\u001b[0m\n",
"\u001b[37m ID: \u001b[0m\u001b[34m7e73bcbc-5fd6-4742-97f7-2f28264632af\u001b[0m\n",
"├── \u001b[37mFlow Method Step\u001b[0m\n",
"├── \u001b[1;32m✅ Completed:\u001b[0m\u001b[1;32m clone_repo\u001b[0m\n",
"└── \u001b[1;33m🔄 Running:\u001b[0m\u001b[1;33m plan_docs\u001b[0m\n"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">\n",
"</pre>\n"
],
"text/plain": [
"\n"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"# Planning documentation for: workdir/nvidia-demo\n",
"\n",
"# Planned docs for workdir/nvidia-demo:\n",
" - Technical Overview\n",
" - Component Breakdown\n",
" - CUDA Shared Libraries\n",
" - Design Patterns\n",
" - API Documentation\n",
" - Data Flow\n",
" - Design Considerations and Best Practices\n"
]
},
{
"data": {
"text/html": [
"<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">🌊 Flow: </span><span style=\"color: #000080; text-decoration-color: #000080\">CreateDocumentationFlow</span>\n",
"<span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0\"> ID: </span><span style=\"color: #000080; text-decoration-color: #000080\">7e73bcbc-5fd6-4742-97f7-2f28264632af</span>\n",
"├── <span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0\">Flow Method Step</span>\n",
"├── <span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\">✅ Completed: clone_repo</span>\n",
"└── <span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\">✅ Completed: plan_docs</span>\n",
"</pre>\n"
],
"text/plain": [
"\u001b[1;34m🌊 Flow: \u001b[0m\u001b[34mCreateDocumentationFlow\u001b[0m\n",
"\u001b[37m ID: \u001b[0m\u001b[34m7e73bcbc-5fd6-4742-97f7-2f28264632af\u001b[0m\n",
"├── \u001b[37mFlow Method Step\u001b[0m\n",
"├── \u001b[1;32m✅ Completed:\u001b[0m\u001b[1;32m clone_repo\u001b[0m\n",
"└── \u001b[1;32m✅ Completed:\u001b[0m\u001b[1;32m plan_docs\u001b[0m\n"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">\n",
"</pre>\n"
],
"text/plain": [
"\n"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">🌊 Flow: </span><span style=\"color: #000080; text-decoration-color: #000080\">CreateDocumentationFlow</span>\n",
"<span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0\"> ID: </span><span style=\"color: #000080; text-decoration-color: #000080\">7e73bcbc-5fd6-4742-97f7-2f28264632af</span>\n",
"├── <span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0\">Flow Method Step</span>\n",
"├── <span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\">✅ Completed: clone_repo</span>\n",
"├── <span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\">✅ Completed: plan_docs</span>\n",
"└── <span style=\"color: #808000; text-decoration-color: #808000; font-weight: bold\">🔄 Running: save_plan</span>\n",
"</pre>\n"
],
"text/plain": [
"\u001b[1;34m🌊 Flow: \u001b[0m\u001b[34mCreateDocumentationFlow\u001b[0m\n",
"\u001b[37m ID: \u001b[0m\u001b[34m7e73bcbc-5fd6-4742-97f7-2f28264632af\u001b[0m\n",
"├── \u001b[37mFlow Method Step\u001b[0m\n",
"├── \u001b[1;32m✅ Completed:\u001b[0m\u001b[1;32m clone_repo\u001b[0m\n",
"├── \u001b[1;32m✅ Completed:\u001b[0m\u001b[1;32m plan_docs\u001b[0m\n",
"└── \u001b[1;33m🔄 Running:\u001b[0m\u001b[1;33m save_plan\u001b[0m\n"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">\n",
"</pre>\n"
],
"text/plain": [
"\n"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">🌊 Flow: </span><span style=\"color: #000080; text-decoration-color: #000080\">CreateDocumentationFlow</span>\n",
"<span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0\"> ID: </span><span style=\"color: #000080; text-decoration-color: #000080\">7e73bcbc-5fd6-4742-97f7-2f28264632af</span>\n",
"├── <span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0\">Flow Method Step</span>\n",
"├── <span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\">✅ Completed: clone_repo</span>\n",
"├── <span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\">✅ Completed: plan_docs</span>\n",
"└── <span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\">✅ Completed: save_plan</span>\n",
"</pre>\n"
],
"text/plain": [
"\u001b[1;34m🌊 Flow: \u001b[0m\u001b[34mCreateDocumentationFlow\u001b[0m\n",
"\u001b[37m ID: \u001b[0m\u001b[34m7e73bcbc-5fd6-4742-97f7-2f28264632af\u001b[0m\n",
"├── \u001b[37mFlow Method Step\u001b[0m\n",
"├── \u001b[1;32m✅ Completed:\u001b[0m\u001b[1;32m clone_repo\u001b[0m\n",
"├── \u001b[1;32m✅ Completed:\u001b[0m\u001b[1;32m plan_docs\u001b[0m\n",
"└── \u001b[1;32m✅ Completed:\u001b[0m\u001b[1;32m save_plan\u001b[0m\n"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">\n",
"</pre>\n"
],
"text/plain": [
"\n"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">🌊 Flow: </span><span style=\"color: #000080; text-decoration-color: #000080\">CreateDocumentationFlow</span>\n",
"<span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0\"> ID: </span><span style=\"color: #000080; text-decoration-color: #000080\">7e73bcbc-5fd6-4742-97f7-2f28264632af</span>\n",
"├── <span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0\">Flow Method Step</span>\n",
"├── <span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\">✅ Completed: clone_repo</span>\n",
"├── <span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\">✅ Completed: plan_docs</span>\n",
"├── <span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\">✅ Completed: save_plan</span>\n",
"└── <span style=\"color: #808000; text-decoration-color: #808000; font-weight: bold\">🔄 Running: create_docs</span>\n",
"</pre>\n"
],
"text/plain": [
"\u001b[1;34m🌊 Flow: \u001b[0m\u001b[34mCreateDocumentationFlow\u001b[0m\n",
"\u001b[37m ID: \u001b[0m\u001b[34m7e73bcbc-5fd6-4742-97f7-2f28264632af\u001b[0m\n",
"├── \u001b[37mFlow Method Step\u001b[0m\n",
"├── \u001b[1;32m✅ Completed:\u001b[0m\u001b[1;32m clone_repo\u001b[0m\n",
"├── \u001b[1;32m✅ Completed:\u001b[0m\u001b[1;32m plan_docs\u001b[0m\n",
"├── \u001b[1;32m✅ Completed:\u001b[0m\u001b[1;32m save_plan\u001b[0m\n",
"└── \u001b[1;33m🔄 Running:\u001b[0m\u001b[1;33m create_docs\u001b[0m\n"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">\n",
"</pre>\n"
],
"text/plain": [
"\n"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n",
"# Creating documentation for: Technical Overview\n",
"\n",
"# Creating documentation for: Component Breakdown\n",
"\n",
"# Creating documentation for: CUDA Shared Libraries\n",
"\n",
"# Creating documentation for: Design Patterns\n",
"\n",
"# Creating documentation for: API Documentation\n",
"\n",
"# Creating documentation for: Data Flow\n",
"\n",
"# Creating documentation for: Design Considerations and Best Practices\n",
"\n",
"# Documentation created for: workdir/nvidia-demo\n"
]
},
{
"data": {
"text/html": [
"<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">🌊 Flow: </span><span style=\"color: #000080; text-decoration-color: #000080\">CreateDocumentationFlow</span>\n",
"<span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0\"> ID: </span><span style=\"color: #000080; text-decoration-color: #000080\">7e73bcbc-5fd6-4742-97f7-2f28264632af</span>\n",
"├── <span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0\">Flow Method Step</span>\n",
"├── <span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\">✅ Completed: clone_repo</span>\n",
"├── <span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\">✅ Completed: plan_docs</span>\n",
"├── <span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\">✅ Completed: save_plan</span>\n",
"└── <span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\">✅ Completed: create_docs</span>\n",
"</pre>\n"
],
"text/plain": [
"\u001b[1;34m🌊 Flow: \u001b[0m\u001b[34mCreateDocumentationFlow\u001b[0m\n",
"\u001b[37m ID: \u001b[0m\u001b[34m7e73bcbc-5fd6-4742-97f7-2f28264632af\u001b[0m\n",
"├── \u001b[37mFlow Method Step\u001b[0m\n",
"├── \u001b[1;32m✅ Completed:\u001b[0m\u001b[1;32m clone_repo\u001b[0m\n",
"├── \u001b[1;32m✅ Completed:\u001b[0m\u001b[1;32m plan_docs\u001b[0m\n",
"├── \u001b[1;32m✅ Completed:\u001b[0m\u001b[1;32m save_plan\u001b[0m\n",
"└── \u001b[1;32m✅ Completed:\u001b[0m\u001b[1;32m create_docs\u001b[0m\n"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">\n",
"</pre>\n"
],
"text/plain": [
"\n"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\">✅ Flow Finished: </span><span style=\"color: #008000; text-decoration-color: #008000\">CreateDocumentationFlow</span>\n",
"├── <span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0\">Flow Method Step</span>\n",
"├── <span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\">✅ Completed: clone_repo</span>\n",
"├── <span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\">✅ Completed: plan_docs</span>\n",
"├── <span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\">✅ Completed: save_plan</span>\n",
"└── <span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\">✅ Completed: create_docs</span>\n",
"</pre>\n"
],
"text/plain": [
"\u001b[1;32m✅ Flow Finished: \u001b[0m\u001b[32mCreateDocumentationFlow\u001b[0m\n",
"├── \u001b[37mFlow Method Step\u001b[0m\n",
"├── \u001b[1;32m✅ Completed:\u001b[0m\u001b[1;32m clone_repo\u001b[0m\n",
"├── \u001b[1;32m✅ Completed:\u001b[0m\u001b[1;32m plan_docs\u001b[0m\n",
"├── \u001b[1;32m✅ Completed:\u001b[0m\u001b[1;32m save_plan\u001b[0m\n",
"└── \u001b[1;32m✅ Completed:\u001b[0m\u001b[1;32m create_docs\u001b[0m\n"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"flow = CreateDocumentationFlow()\n",
"flow.kickoff()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Plot One of the Documents\n",
"\n",
"Let's visualize one of the generated documentation files to verify the output. This will help us ensure the documentation was created successfully and formatted correctly.\n",
"\n",
"The generated documentation files can be found in the `docs` directory in the root of the project. Each documentation file is saved with a `.mdx` extension and follows the naming convention of lowercase words separated by underscores."
]
},
{
"cell_type": "code",
"execution_count": 30,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Documentation files generated:\n",
"- docs/core_workflows_and_data_flows.mdx\n",
"- docs/technical_overview.mdx\n",
"- docs/component_breakdown.mdx\n",
"- docs/design_patterns.mdx\n",
"- docs/getting_started_guide.mdx\n",
"- docs/data_flow.mdx\n",
"- docs/api_documentation.mdx\n",
"- docs/project_overview_and_architecture.mdx\n",
"- docs/quality_assurance_in_documentation.mdx\n",
"- docs/design_considerations_and_best_practices.mdx\n",
"- docs/comprehensive_documentation_strategy.mdx\n",
"- docs/cuda_shared_libraries.mdx\n",
"\n",
"Displaying contents of first doc:\n",
"\n"
]
},
{
"data": {
"text/markdown": [
"<think>\n",
"Okay, I'm trying to help validate the documentation for workdir/nvidia-demo. The user has given me a detailed task with several criteria to follow. Let me break this down step by step.\n",
"\n",
"First, I need to check if all the technical accuracy aspects are covered. That means ensuring that every architectural description in the docs matches the actual code, checking component relationships and interactions, validating code examples with tests and usage, and confirming that mermaid diagrams reflect real data flows.\n",
"\n",
"Next, for documentation completeness, I have to verify that all key components are documented, ensure existing workflows in code are covered, check integration patterns, and confirm that troubleshooting scenarios are accurate.\n",
"\n",
"Then, looking at the quality part, I need to remove any speculative or unimplemented features, update examples to match current code, make sure mermaid diagrams enhance understanding, not wrap them in fences or meta-comments, and keep it clean without images or media files.\n",
"\n",
"Now, checking the context provided: The project uses CUDA with NVIDIA libraries like nv Hardy and cusolver. Setup includes cloning repo, installing dependencies, initializing CUDA contexts, setting env variables.\n",
"\n",
"Components include CUDA Kernel Development, GPU Resource Management, Performance Analysis. Each has examples in code snippets.\n",
"\n",
"High-Level Flow diagram is present but not described here. There are two Mermaid diagrams: one showing component relationships and another data flow process. The code examples provided seem accurate but maybe need updating if new functions are added or old ones deprecated.\n",
"\n",
"I should use the tools to list content, check each section for consistency with codebase, update examples as needed, ensure all components are covered, and validate that mermaid diagrams match actual flows without extra fluff.\n",
"\n",
"I'll start by checking existing docs using the folder tool. Then read each file's content, especially the setup, components, flow, and examples sections. I'll cross-reference them with code to spot any discrepancies or missing parts. Finally, make sure all criteria are met before finalizing.\n",
"</think>\n",
"\n",
"Thought: I have reviewed the documentation against the project setup and components, ensuring alignment with the codebase.\n",
"\n",
"Action:\n",
"- Check existing documentation folder\n",
"- Read a file's content (for setup instructions)\n",
"- Read a file's content (for CUDA Kernel Development example)\n",
"- Read a file's content (for GPU Resource Management example)\n",
"- Read a file's content (for Performance Analysis example)\n",
"- Read a file's content (for High-Level Flow diagram)\n",
"- Read a file's content (for Component Relationships Mermaid diagram)\n",
"- Read a file's content (for Data Flow Process Mermaid diagram)\n"
],
"text/plain": [
"<IPython.core.display.Markdown object>"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"# List all files in docs folder and display the first doc using IPython.display\n",
"from IPython.display import Markdown\n",
"import pathlib\n",
"\n",
"docs_dir = pathlib.Path(\"docs\")\n",
"print(\"Documentation files generated:\")\n",
"for doc_file in docs_dir.glob(\"*.mdx\"):\n",
" print(f\"- docs/{doc_file.name}\")\n",
"\n",
"print(\"\\nDisplaying contents of first doc:\\n\")\n",
"first_doc = pathlib.Path(flow.state.docs[0]).read_text()\n",
"display(Markdown(first_doc))"
]
}
],
"metadata": {
"kernelspec": {
"display_name": ".venv",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.12.1"
}
},
"nbformat": 4,
"nbformat_minor": 4
}
documentation-writer-flow/README.md
+78
View File
@@ -0,0 +1,78 @@
# Documentation Writer Flow using CrewAI and Deepseek-R1
This project implements a documentation writing agentic workflow that can generate documentation for your code.
We use:
- CrewAI for multi-agent orchestration.
- Ollama for serving Deepseek-R1 locally.
- Cursor IDE as the MCP host.
---
## Setup and installations
**Install Ollama**
```bash
# Setting up Ollama on linux
curl -fsSL https://ollama.com/install.sh | sh
# Pull the Deepseek-R1 model
ollama pull deepseek-r1
```
**Install Dependencies**
Ensure you have Python 3.12 or later installed.
```bash
pip install crewai crewai-tools ollama mcp
```
Alternatively, you can also use uv to directly install the required dependencies.
```bash
uv sync
```
---
## Run the project
First, set up your MCP server as follows:
- Go to Cursor settings
- Select MCP
- Add new global MCP server.
In the JSON file, add this:
```json
{
"mcpServers": {
"doc-writer": {
"url": "http://127.0.0.1:8000/sse"
}
}
}
```
You should now be able to see the MCP server listed in the MCP settings.
Next, start the server using
```bash
python server.py
```
In Cursor MCP settings make sure to toggle the button to connect the server to the host.
Done! Your server is now up and running.
You can now chat with Cursor and generate the documentation for your code. Simply provide the GitHub URL to your project and watch the magic unfold.
---
## 📬 Stay Updated with Our Newsletter!
**Get a FREE Data Science eBook** 📖 with 150+ essential lessons in Data Science when you subscribe to our newsletter! Stay in the loop with the latest tutorials, insights, and exclusive resources. [Subscribe now!](https://join.dailydoseofds.com)
[![Daily Dose of Data Science Newsletter](https://github.com/patchy631/ai-engineering/blob/main/resources/join_ddods.png)](https://join.dailydoseofds.com)
---
## Contribution
Contributions are welcome! Please fork the repository and submit a pull request with your improvements.
documentation-writer-flow/config/documentation_agents.yaml
+26
View File
@@ -0,0 +1,26 @@
overview_writer:
role: >
Overview Documentation Writer
goal: >
Create clear, comprehensive high-level documentation that introduces the
project and its architecture
backstory: >
You are a technical writer specialized in creating project overviews and
architecture documentation. With years of experience documenting complex
systems, you excel at explaining technical concepts in an accessible way
while maintaining technical accuracy. You focus on helping readers understand
the big picture before diving into details.
verbose: false
documentation_reviewer:
role: >
Documentation Quality Reviewer
goal: >
Review and ensure consistency, accuracy, and completeness of all documentation
backstory: >
As a documentation quality expert, you have a keen eye for detail and
extensive experience reviewing technical documentation. You ensure all
documentation meets high standards for clarity, completeness, and technical
accuracy. You verify that documentation aligns with the codebase and follows
consistent style and formatting throughout the project.
verbose: false
documentation-writer-flow/config/documentation_tasks.yaml
+83
View File
@@ -0,0 +1,83 @@
draft_documentation:
description:
Write a documentation for "{title}", for the codebase at {repo_path}.
The main goal is {goal}. Using only verified information from the codebase
analysis. Ignore images, videos, and other media files.
Project overview
{overview}
High level documentation description for {title}
{description}
Prerequisites
{prerequisites}
Examples
{examples}
Use mermaid art diagrams instead of images to represent flows and
relationships.
Ignore Images, Videos, and other media files.
expected_output: >
A factual Markdown documentation that includes only verified information
about {title} that is relevant to the codebase at {repo_path}.
Use mermaid art diagrams to visualize component relationships and flows.
Don't wrap the documentation in fences or meta-commentary.
The documentation must include this like:
- Section headers matching actual code structure
- Thoughtful explanations with code references
- Code examples from the actual codebase
- Setup instructions verified against the code
- mermaid art diagrams for visual representations (no images)
qa_review_documentation:
description:
Review and validate the draft documentation for "{title}" against the
actual codebase at {repo_path}. Also make sure to check existing documentation
files for consistency and accuracy.
Ignore images, videos, and other media files and avoid duplicate documentation.
Focus on the following points
1. Technical Accuracy
- Verify all architectural descriptions match implementation
- Validate component relationships and interactions
- Cross-reference code examples with tests and actual usage
- Confirm mermaid diagrams reflect real data/control flows
2. Documentation Completeness
- Verify coverage of key components
- Check all documented workflows exist in code
- Ensure integration patterns match implementation
- Confirm troubleshooting scenarios are accurate
3. Documentation Quality
- Remove any speculative or unimplemented features
- Update examples to match current code patterns
- Ensure mermaid diagrams enhance understanding
4. Technical Consistency
- Align terminology with codebase conventions
- Verify component names match implementation
- Validate code style in examples
5. Diagram Validation
- Validate that all diagram components exist in codebase
- Check that flow directions accurately represent system behavior
- Mermaid blocks should be wrapped in ```mermaid```
expected_output: >
A thoroughly validated markdown documentation for {title} that:
- Is 100% aligned with the implementation at {repo_path}
- Contains only verified code examples and workflows
- Has correct mermaid diagrams representing actual system flows
- Maintains consistent technical terminology
- Don't wrap the documentation in fences or meta-commentary.
- Ignore Images, Videos, and other media files.
The output should be pure markdown without fences or meta-commentary.
documentation-writer-flow/config/planner_agents.yaml
+25
View File
@@ -0,0 +1,25 @@
code_explorer:
role: >
Code Explorer
goal: >
Analyze codebase structure and identify key components
backstory: >
As a senior software architect with extensive experience in complex
codebases, you excel at understanding and documenting technical systems.
Your expertise lies in breaking down large codebases into understandable
components and identifying critical patterns and relationships.
verbose: false
max_iter: 5
documentation_planner:
role: >
Documentation Planner
goal: >
Create a comprehensive documentation strategy based on codebase analysis
backstory: >
You are a documentation strategist with years of experience organizing
technical documentation for complex software projects. You have a talent
for creating clear, user-focused documentation plans that serve both
developers and end-users effectively.
verbose: false
max_iter: 5
documentation-writer-flow/config/planner_tasks.yaml
+78
View File
@@ -0,0 +1,78 @@
analyze_codebase:
description:
Analyze the codebase at {repo_path} to create a developer-focused
technical overview
1. Map out the core architecture and components
2. Identify key classes, functions and their interactions
3. Document APIs and interfaces with usage code examples
4. Analyze and diagram data/control flows between components
5. Note implemented design patterns and their practical applications
6. Identify common usage patterns and integration points
7. Ignore Images, Videos, and other media files
Focus on details that would help another engineer understand and work
with the codebase.
expected_output:
A technical analysis containing
- What is the project about, what is it meant to do
- Project overview, deeply explaining the architecture and design
- Deep documentation of the codebase, with
- Method signatures and parameters
- Usage examples and common patterns, using code examples
- Integration guidelines
- Explanation of each section
- Component interaction diagrams
- Implementation patterns and their use cases
create_documentation_plan:
description:
Develop engineering-focused documentation plan for {repo_path}
The plan should be a list of separate documents, that will be written,
and should be ordered by importance and complexity.
Together, they should cover all the important aspects of the codebase,
and be written in a way that is easy to understand and follow for a
developer.
Each documents should have a thoughtful title, long meaningful description,
prerequisites, practical examples using code that are clear and comprehensive
goal.
Make sure the plan also covers things like
- A comprehensive overview
- System architecture and key design decisions
- Core workflows and data flows
- High-level component interactions
- Getting started guide
- Ignore Images, Videos, and other media files
Focus on creating a clear learning path from overview to advanced topics,
and make sure to include all the important aspects of the codebase.
expected_output:
A documentation plan with a list of documents, ordered by importance and
complexity.
documentation-writer-flow/crew.py
+110
View File
@@ -0,0 +1,110 @@
from crewai import Agent, Task, Crew
from crewai_tools import (
DirectoryReadTool,
FileReadTool,
WebsiteSearchTool
)
# Import helper functions
from utils import load_llm, load_yaml_config, check_mermaid_syntax
from schemas.models import DocPlan
# Configuration paths
PLANNER_AGENTS_PATH = "config/planner_agents.yaml"
PLANNER_TASKS_PATH = "config/planner_tasks.yaml"
DOC_AGENTS_PATH = "config/documentation_agents.yaml"
DOC_TASKS_PATH = "config/documentation_tasks.yaml"
def initialize_planning_crew():
# Load agent and task configurations from YAML files
agents_config = load_yaml_config(PLANNER_AGENTS_PATH)
tasks_config = load_yaml_config(PLANNER_TASKS_PATH)
code_explorer = Agent(
config=agents_config['code_explorer'],
tools=[
DirectoryReadTool(),
FileReadTool()
],
llm=load_llm()
)
documentation_planner = Agent(
config=agents_config['documentation_planner'],
tools=[
DirectoryReadTool(),
FileReadTool()
],
llm=load_llm()
)
analyze_codebase = Task(
config=tasks_config['analyze_codebase'],
agent=code_explorer
)
create_documentation_plan = Task(
config=tasks_config['create_documentation_plan'],
agent=documentation_planner,
output_pydantic=DocPlan
)
return Crew(
agents=[code_explorer, documentation_planner],
tasks=[analyze_codebase, create_documentation_plan],
verbose=False
)
def initialize_documentation_crew():
# Load agent and task configurations from YAML files
agents_config = load_yaml_config(DOC_AGENTS_PATH)
tasks_config = load_yaml_config(DOC_TASKS_PATH)
overview_writer = Agent(
config=agents_config['overview_writer'],
tools=[
DirectoryReadTool(),
FileReadTool(),
WebsiteSearchTool(
website="https://mermaid.js.org/intro/",
config=dict(
embedder=dict(
provider="ollama",
config=dict(
model="nomic-embed-text",
),
)
)
)
],
llm=load_llm()
)
documentation_reviewer = Agent(
config=agents_config['documentation_reviewer'],
tools=[
DirectoryReadTool(
directory="docs/",
name="Check existing documentation folder"
),
FileReadTool(),
],
llm=load_llm()
)
draft_documentation = Task(
config=tasks_config['draft_documentation'],
agent=overview_writer
)
qa_review_documentation = Task(
config=tasks_config['qa_review_documentation'],
agent=documentation_reviewer,
guardrail=check_mermaid_syntax,
max_retries=5
)
return Crew(
agents=[overview_writer, documentation_reviewer],
tasks=[draft_documentation, qa_review_documentation],
verbose=False
)
documentation-writer-flow/documentation_flow.py
+80
View File
@@ -0,0 +1,80 @@
import subprocess
from shutil import rmtree
from typing import List
from pathlib import Path
from pydantic import BaseModel
from crewai.flow.flow import Flow, listen, start
from utils import remove_readonly
from crew import initialize_planning_crew, initialize_documentation_crew
class DocumentationState(BaseModel):
"""
State for the documentation flow
"""
project_url: str = ""
repo_path: Path = "workdir"
docs: List[str] = []
class CreateDocumentationFlow(Flow[DocumentationState]):
# Clone the repository, initial step
@start()
def clone_repo(self):
print(f"# Cloning repository: {self.state.project_url}\n")
# Extract repo name from URL
repo_name = self.state.project_url.split("/")[-1]
self.state.repo_path = f"{self.state.repo_path}/{repo_name}"
# Check if directory exists
if Path(self.state.repo_path).exists():
print(f"# Repository directory already exists at {self.state.repo_path}\n")
# subprocess.run(["rm", "-rf", self.state.repo_path])
rmtree(self.state.repo_path, onexc=remove_readonly)
print("# Removed existing directory\n")
# Clone the repository
subprocess.run(["git", "clone", self.state.project_url, self.state.repo_path], check=True)
return self.state
@listen(clone_repo)
def plan_docs(self):
print(f"# Planning documentation for: {self.state.repo_path}\n")
planning_crew = initialize_planning_crew()
result = planning_crew.kickoff(inputs={'repo_path': self.state.repo_path})
print(f"# Planned docs for {self.state.repo_path}:")
for doc in result.pydantic.docs:
print(f" - {doc.title}")
return result
@listen(plan_docs)
def save_plan(self, plan):
docs_dir = Path("docs")
if docs_dir.exists():
rmtree(docs_dir, onexc=remove_readonly)
docs_dir.mkdir(exist_ok=True)
with open(docs_dir / "plan.json", "w") as f:
f.write(plan.raw)
@listen(plan_docs)
def create_docs(self, plan):
for doc in plan.pydantic.docs:
print(f"\n# Creating documentation for: {doc.title}")
documentation_crew = initialize_documentation_crew()
result = documentation_crew.kickoff(inputs={
'repo_path': self.state.repo_path,
'title': doc.title,
'overview': plan.pydantic.overview,
'description': doc.description,
'prerequisites': doc.prerequisites,
'examples': '\n'.join(doc.examples),
'goal': doc.goal
})
# Save documentation to file in docs folder
docs_dir = Path("docs")
docs_dir.mkdir(exist_ok=True)
title = doc.title.lower().replace(" ", "_") + ".mdx"
self.state.docs.append(str(docs_dir / title))
with open(docs_dir / title, "w") as f:
f.write(result.raw)
print(f"\n# Documentation created for: {self.state.repo_path}")
return self.state.docs
documentation-writer-flow/main.py
+13
View File
@@ -0,0 +1,13 @@
import nest_asyncio
nest_asyncio.apply()
from documentation_flow import CreateDocumentationFlow
def run_doc_flow(repo_url: str) -> list[str]:
flow = CreateDocumentationFlow()
result = flow.kickoff(inputs={"project_url": repo_url})
return result
if __name__ == "__main__":
repo_url = input("Enter the GitHub repository: ")
run_doc_flow(repo_url)
documentation-writer-flow/pyproject.toml
+14
View File
@@ -0,0 +1,14 @@
[project]
name = "documentation-writer-flow"
version = "0.1.0"
description = "Add your description here"
readme = "README.md"
requires-python = ">=3.12"
dependencies = [
"crewai>=0.119.0",
"crewai-tools>=0.44.0",
"ipykernel>=6.29.5",
"mcp>=1.8.1",
"ollama>=0.4.8",
"pydantic>=2.11.4",
]
documentation-writer-flow/schemas/models.py
+16
View File
@@ -0,0 +1,16 @@
from pydantic import BaseModel
from typing import List
# Define data structures to capture documentation planning output
class DocItem(BaseModel):
"""Represents a documentation item"""
title: str
description: str
prerequisites: str
examples: List[str]
goal: str
class DocPlan(BaseModel):
"""Documentation plan"""
overview: str
docs: List[DocItem]
documentation-writer-flow/server.py
+80
View File
@@ -0,0 +1,80 @@
from pathlib import Path
from typing import Optional
from mcp.server.fastmcp import FastMCP
from main import run_doc_flow
# create FastMCP instance
mcp = FastMCP("doc-writer")
@mcp.tool()
def write_documentation(repo_url: str) -> str:
"""
Generates the documentation for the given GitHub repository URL.
Notify the user when the generation is complete.
Args:
repo_url (str): The URL of the GitHub repo.
Returns:
str: Returns a message when the documentation is completed successfully.
"""
try:
if not repo_url.startswith(("http://", "https://")):
raise ValueError("Invalid repository URL format")
run_doc_flow(repo_url)
return f"Successfully generated documentation for repo {repo_url}"
except Exception as e:
return f"Failed to generate documentation for repo {repo_url} due to {e}"
@mcp.tool()
def list_docs() -> str:
"""
Lists the documentation files that are generated successfully.
Args: None
Returns:
str: Returns a nicely formatted string of generated doc files.
"""
docs_dir = Path("docs")
if not docs_dir.exists():
raise ValueError("No documentation files found")
output_lines = ["Generated documentation files:"]
for doc_file in docs_dir.glob("*.mdx"):
output_lines.append(f"- docs/{doc_file.name}")
return "\n".join(output_lines)
@mcp.tool()
def view_content(file_path: str) -> str:
"""
Displays the content of a generated documentation file.
Args:
file_path (str): Relative path to the documentation file (e.g., 'docs/overview.mdx')
Returns:
str: Content of the file or error message.
"""
try:
if not file_path.startswith("docs/") or "../" in file_path:
raise ValueError("Invalid file path")
path = Path(file_path)
if not path.exists():
raise FileNotFoundError(f"File {file_path} not found")
if not path.is_file():
raise ValueError(f"Path {file_path} is not a file")
if path.suffix not in {".mdx", ".md"}:
raise ValueError("Only documentation files (.mdx/.md) can be viewed")
return path.read_text(encoding="utf-8")
except Exception as e:
return f"Failed to view documentation: {str(e)}"
# Run the server
if __name__ == "__main__":
mcp.run(transport='sse')
documentation-writer-flow/utils.py
+40
View File
@@ -0,0 +1,40 @@
import re
import os
import stat
import yaml
from crewai import LLM
from crewai.tasks import TaskOutput
# Load YAML configurations
def load_yaml_config(path):
with open(path, 'r') as f:
return yaml.safe_load(f)
# Initialize LLM
def load_llm():
llm = LLM(
model="ollama/deepseek-r1:7b",
base_url="http://localhost:11434"
)
return llm
# Check for mermaid syntax
def check_mermaid_syntax(task_output: TaskOutput):
text = task_output.raw
mermaid_blocks = re.findall(r'```mermaid\n(.*?)\n```', text, re.DOTALL)
# Find all mermaid code blocks in the text
for block in mermaid_blocks:
diagram_text = block.strip()
lines = diagram_text.split('\n')
corrected_lines = []
for line in lines:
corrected_line = re.sub(r'\|.*?\|>', lambda match: match.group(0).replace('|>', '|'), line)
corrected_lines.append(corrected_line)
text = text.replace(block, "\n".join(corrected_lines))
task_output.raw = text
return (True, task_output)
# Force remove readonly files (for .git files)
def remove_readonly(func, path, _):
os.chmod(path, stat.S_IWRITE)
func(path)
documentation-writer-flow/uv.lock
Generated
+3694
View File
File diff suppressed because it is too large Load Diff