chore: import upstream snapshot with attribution
This commit is contained in:
wehub-resource-sync
@@ -0,0 +1,3 @@
|
||||
FIRECRAWL_API_KEY=your_firecrawl_api_key
|
||||
OPENAI_API_KEY=your_openai_api_key_incase_you_want_to_use_openai_for_llm
|
||||
TYPEFULLY_API_KEY=your_typefully_api_key
|
||||
@@ -0,0 +1,34 @@
|
||||
|
||||
# Content writing agentic-workflow
|
||||
|
||||
This project leverages CrewAI Flow to scrape a website, prepare a social post and publish it, powered by a locally running Llama 3.2!
|
||||
|
||||
### Watch Demo on YouTube
|
||||
[](https://www.youtube.com/watch?v=Nor6vNl1NPo)
|
||||
|
||||
|
||||
## Installation and setup
|
||||
|
||||
**Get API Keys**:
|
||||
- [FireCrawl](https://docs.firecrawl.dev/introduction)
|
||||
- [Typefully](https://support.typefully.com/en/articles/8718287-typefully-api)
|
||||
|
||||
|
||||
**Install Dependencies**:
|
||||
Ensure you have Python 3.11 or later installed.
|
||||
```bash
|
||||
pip install crewai crewai-tools
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 📬 Stay Updated with Our Newsletter!
|
||||
**Get a FREE Data Science eBook** 📖 with 150+ essential lessons in Data Science when you subscribe to our newsletter! Stay in the loop with the latest tutorials, insights, and exclusive resources. [Subscribe now!](https://join.dailydoseofds.com)
|
||||
|
||||
[](https://join.dailydoseofds.com)
|
||||
|
||||
---
|
||||
|
||||
## Contribution
|
||||
|
||||
Contributions are welcome! Please fork the repository and submit a pull request with your improvements.
|
||||
@@ -0,0 +1,35 @@
|
||||
draft_analyzer:
|
||||
role: >
|
||||
Draft Analyzer
|
||||
goal: >
|
||||
Analyze draft and identify key ideas, sections and technical concepts
|
||||
backstory: >
|
||||
You are a technical writer with years of experience writing, editing and
|
||||
reviewing technical blogs. You have a talent for understanding and
|
||||
documenting technical concepts.
|
||||
verbose: false
|
||||
|
||||
twitter_thread_planner:
|
||||
role: >
|
||||
Twitter Thread Planner
|
||||
goal: >
|
||||
Create a Twitter thread plan based on the provided draft analysis
|
||||
backstory: >
|
||||
You are a technical writer with years of experience in converting long
|
||||
technical blogs into Twitter threads. You have a talent for breaking longform
|
||||
content into bite-sized tweets that are engaging and informative. And identify
|
||||
relevant urls to media that can be associated with a tweet.
|
||||
verbose: false
|
||||
|
||||
linkedin_post_planner:
|
||||
role: >
|
||||
LinkedIn Post Planner
|
||||
goal: >
|
||||
Create an engaging LinkedIn post based on the provided draft analysis
|
||||
backstory: >
|
||||
You are a technical writer with extensive experience crafting technical LinkedIn
|
||||
content. You excel at distilling technical concepts into clear, authoritative posts
|
||||
that resonate with a professional audience while maintaining technical accuracy.
|
||||
You know how to balance technical depth with accessibility and incorporate relevant
|
||||
hashtags and mentions to maximize engagement.
|
||||
verbose: false
|
||||
@@ -0,0 +1,83 @@
|
||||
analyze_draft:
|
||||
description: |
|
||||
Analyze the markdown file at {draft_path} to create a developer-focused
|
||||
technical overview
|
||||
|
||||
1. Map out the core idea that the blog discusses
|
||||
|
||||
2. Identify key sections and what each section is about
|
||||
|
||||
3. For each section, extract all URLs that appear inside image markdown syntax 
|
||||
|
||||
4. You must associate these identified image urls to their correspoinding sections, so that we can use them with the tweets as media pieces
|
||||
|
||||
Focus on details that are important for a comprehensive understanding of
|
||||
the blog.
|
||||
expected_output: |
|
||||
A technical analysis containing:
|
||||
- Blog title and core concept/idea
|
||||
- Key technical sections identified with their main points
|
||||
- Important code examples or technical concepts covered
|
||||
- Key takeaways for developers
|
||||
- Relevant urls to media that are associated with the key sections and can be associated with a tweet, this must be done
|
||||
|
||||
create_twitter_thread_plan:
|
||||
description: |
|
||||
Develop an engaging Twitter thread based on the draft analysis provided and closely follow the writing style prvided in the {path_to_example_threads}
|
||||
|
||||
The thread should break down complex technical concepts into digestible, tweet-sized chunks
|
||||
that maintain technical accuracy while being accessible.
|
||||
|
||||
Plan should include:
|
||||
- A strong hook tweet that captures attention, it should be under 10 words, it must be same as the title of the blog
|
||||
- Logical flow from basic to advanced concepts
|
||||
- Code snippets or key technical highlights that fit Twitter's format
|
||||
- Relevant urls to media that are associated with the key sections and must be associated with their corresponding tweets
|
||||
- Clear takeaways for engineering audience
|
||||
|
||||
Make sure to cover:
|
||||
- close follow the writing style provided in the {path_to_example_threads}
|
||||
- The core problem being solved
|
||||
- Key technical innovations or approaches
|
||||
- Interesting implementation details
|
||||
- Real-world applications or benefits
|
||||
- Call to action for the conclusion
|
||||
- Add relevant urls to each tweet that can be associated with a tweet
|
||||
|
||||
Focus on creating a narrative that technical audiences will find valuable
|
||||
while keeping each tweet concise, accessible and impactful.
|
||||
expected_output: |
|
||||
A Twitter thread with a list of tweets, where each tweet has the following:
|
||||
- content
|
||||
- urls to media that are associated with the tweet, whenever possible
|
||||
- is_hook: true if the tweet is a hook tweet, false otherwise
|
||||
|
||||
|
||||
create_linkedin_post_plan:
|
||||
description: |
|
||||
Develop a comprehensive LinkedIn post based on the draft analysis provided
|
||||
|
||||
The post should present technical content in a professional, long-form format
|
||||
while maintaining engagement and readability.
|
||||
|
||||
Plan should include:
|
||||
- An attention-grabbing opening statement, it should be same as the title of the blog
|
||||
- Well-structured body that breaks down the technical content
|
||||
- Professional tone suitable for LinkedIn's business audience
|
||||
- One main blog URL placed strategically at the end of the post
|
||||
- Strategic use of line breaks and formatting
|
||||
- Relevant hashtags (3-5 maximum)
|
||||
|
||||
Make sure to cover:
|
||||
- The core technical problem and its business impact
|
||||
- Key solutions and technical approaches
|
||||
- Real-world applications and benefits
|
||||
- Professional insights or lessons learned
|
||||
- Clear call to action
|
||||
|
||||
Focus on creating content that resonates with both technical professionals
|
||||
and business leaders while maintaining technical accuracy.
|
||||
expected_output: |
|
||||
A LinkedIn post plan containing:
|
||||
- content
|
||||
- a main blog url that is associated with the post
|
||||
@@ -0,0 +1,534 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Content Planning and Publishing Crew"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"This notebook demonstrates how to create an AI crew for planning and publishing content using CrewAI Flows.\n",
|
||||
"The crew will take a link to blog post, download content as markdown using firecrawl, analyze it and generate a twitter thread and schedule it on Typefully."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Initialization and Setup\n",
|
||||
"Initial imports for the CrewAI Flow and Crew and setting up the environment"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 20,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Importing necessary libraries\n",
|
||||
"import getpass\n",
|
||||
"import os\n",
|
||||
"import datetime\n",
|
||||
"import uuid\n",
|
||||
"import yaml\n",
|
||||
"import json\n",
|
||||
"import subprocess\n",
|
||||
"from pathlib import Path\n",
|
||||
"import pydantic\n",
|
||||
"from pydantic import BaseModel\n",
|
||||
"from typing import Optional\n",
|
||||
"\n",
|
||||
"# Firecrawl SDK\n",
|
||||
"from firecrawl import FirecrawlApp\n",
|
||||
"\n",
|
||||
"# Typefully scheduler\n",
|
||||
"import scheduler\n",
|
||||
"\n",
|
||||
"# Importing Crew related components\n",
|
||||
"from crewai import Agent, Task, Crew, LLM\n",
|
||||
"\n",
|
||||
"# Importing CrewAI Flow related components\n",
|
||||
"from crewai.flow.flow import Flow, listen, start, router, or_\n",
|
||||
"\n",
|
||||
"from dotenv import load_dotenv\n",
|
||||
"load_dotenv()\n",
|
||||
"# Apply a patch to allow nested asyncio loops in Jupyter\n",
|
||||
"import nest_asyncio\n",
|
||||
"nest_asyncio.apply()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Setup LLM\n",
|
||||
"\n",
|
||||
"Make sure you have ollama installed and running on your machine"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"llm = LLM(\n",
|
||||
" model=\"ollama/llama3.2\",\n",
|
||||
" base_url=\"http://localhost:11434\"\n",
|
||||
")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Blog Post URL"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 21,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"blog_post_url = \"https://blog.dailydoseofds.com/p/5-chunking-strategies-for-rag\""
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Plan for our Flow\n",
|
||||
"\n",
|
||||
"1. Scrape the blog post\n",
|
||||
"2. Decode where to post using a router\n",
|
||||
"3. Kickoff the right **[Crew of Agents]** to prepare a draft ready to publish\n",
|
||||
"4. Publish it using typefully"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 22,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/html": [
|
||||
"<img src=\"content_writing_flow.png\" width=\"1000\" height=\"750\"/>"
|
||||
],
|
||||
"text/plain": [
|
||||
"<IPython.core.display.HTML object>"
|
||||
]
|
||||
},
|
||||
"execution_count": 22,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"from IPython.display import HTML\n",
|
||||
"HTML('<img src=\"content_writing_flow.png\" width=\"1000\" height=\"750\"/>')"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Twitter Thread Planning Crew\n",
|
||||
"\n",
|
||||
"This structure will be used to capture the output of the planning crew which will be used to create the twitter thread and schedule it on Typefully."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 27,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"class Tweet(BaseModel):\n",
|
||||
" \"\"\"Represents an individual tweet in a thread\"\"\"\n",
|
||||
" content: str\n",
|
||||
" is_hook: bool = False # Identifies if this is the opening/hook tweet\n",
|
||||
" media_urls: Optional[list[str]] = [] # Optional media attachments (images, code snippets)\n",
|
||||
"\n",
|
||||
"class Thread(BaseModel):\n",
|
||||
" \"\"\"Represents a Twitter thread\"\"\"\n",
|
||||
" topic: str # Main topic/subject of the thread\n",
|
||||
" tweets: list[Tweet] # List of tweets in the thread"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 28,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from crewai_tools import (\n",
|
||||
" DirectoryReadTool,\n",
|
||||
" FileReadTool,\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"# Load agent and task configurations from YAML files\n",
|
||||
"with open('config/planner_agents.yaml', 'r') as f:\n",
|
||||
" agents_config = yaml.safe_load(f)\n",
|
||||
"\n",
|
||||
"with open('config/planner_tasks.yaml', 'r') as f:\n",
|
||||
" tasks_config = yaml.safe_load(f)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 32,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stderr",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"2024-12-19 14:18:07,847 - 7988681536 - __init__.py-__init__:537 - WARNING: Overriding of current TracerProvider is not allowed\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"draft_analyzer = Agent(config=agents_config['draft_analyzer'], tools=[\n",
|
||||
" DirectoryReadTool(),\n",
|
||||
" FileReadTool()\n",
|
||||
"], llm=llm)\n",
|
||||
"twitter_thread_planner = Agent(config=agents_config['twitter_thread_planner'], tools=[\n",
|
||||
" DirectoryReadTool(),\n",
|
||||
" FileReadTool()\n",
|
||||
"], llm=llm)\n",
|
||||
"\n",
|
||||
"analyze_draft = Task(\n",
|
||||
" config=tasks_config['analyze_draft'],\n",
|
||||
" agent=draft_analyzer\n",
|
||||
")\n",
|
||||
"create_twitter_thread_plan = Task(\n",
|
||||
" config=tasks_config['create_twitter_thread_plan'],\n",
|
||||
" agent=twitter_thread_planner,\n",
|
||||
" output_pydantic=Thread\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"planning_crew = Crew(\n",
|
||||
" agents=[draft_analyzer, twitter_thread_planner],\n",
|
||||
" tasks=[analyze_draft, create_twitter_thread_plan],\n",
|
||||
" verbose=False\n",
|
||||
")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# LinkedIn Post Planning Crew"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 38,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"class LinkedInPost(BaseModel):\n",
|
||||
" \"\"\"Represents a LinkedIn post\"\"\"\n",
|
||||
" content: str\n",
|
||||
" media_url: str # Main image url for the post"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 39,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stderr",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"2024-12-19 14:25:39,453 - 7988681536 - __init__.py-__init__:537 - WARNING: Overriding of current TracerProvider is not allowed\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"linkedin_post_planner = Agent(config=agents_config['linkedin_post_planner'], llm=llm)\n",
|
||||
"\n",
|
||||
"create_linkedin_post_plan = Task(\n",
|
||||
" config=tasks_config['create_linkedin_post_plan'],\n",
|
||||
" agent=linkedin_post_planner,\n",
|
||||
" output_pydantic=LinkedInPost\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"linkedin_planning_crew = Crew(\n",
|
||||
" agents=[draft_analyzer, linkedin_post_planner],\n",
|
||||
" tasks=[analyze_draft, create_linkedin_post_plan],\n",
|
||||
" verbose=False\n",
|
||||
")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Create Content Planning Flow\n",
|
||||
"\n",
|
||||
"A Flow to create the content planning for twitter and linkedin using separate crews for twitter and linkedin"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 37,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/html": [
|
||||
"<img src=\"content_writing_flow.png\" width=\"1000\" height=\"750\"/>"
|
||||
],
|
||||
"text/plain": [
|
||||
"<IPython.core.display.HTML object>"
|
||||
]
|
||||
},
|
||||
"execution_count": 37,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"from IPython.display import HTML\n",
|
||||
"HTML('<img src=\"content_writing_flow.png\" width=\"1000\" height=\"750\"/>')"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 40,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from crewai.flow.flow import Flow, listen, start, router, or_\n",
|
||||
"\n",
|
||||
"class ContentPlanningState(BaseModel):\n",
|
||||
" \"\"\"\n",
|
||||
" State for the content planning flow\n",
|
||||
" \"\"\"\n",
|
||||
" blog_post_url: str = blog_post_url\n",
|
||||
" draft_path: Path = \"workdir/\"\n",
|
||||
" post_type: str = \"twitter\"\n",
|
||||
" path_to_example_threads: str = \"workdir/example_threads.txt\"\n",
|
||||
"\n",
|
||||
"class CreateContentPlanningFlow(Flow[ContentPlanningState]):\n",
|
||||
" # Scrape the blog post \n",
|
||||
" # No need for AI Agents on this step, so we just use regular Python code\n",
|
||||
" @start()\n",
|
||||
" def scrape_blog_post(self):\n",
|
||||
" print(f\"# fetching draft from: {self.state.blog_post_url}\")\n",
|
||||
" app = FirecrawlApp(api_key=os.getenv(\"FIRECRAWL_API_KEY\"))\n",
|
||||
" scrape_result = app.scrape_url(self.state.blog_post_url, params={'formats': ['markdown', 'html']})\n",
|
||||
" try:\n",
|
||||
" title = scrape_result['metadata']['title']\n",
|
||||
" except Exception as e:\n",
|
||||
" title = str(uuid.uuid4())\n",
|
||||
" self.state.draft_path = f'workdir/{title}.md'\n",
|
||||
" with open(self.state.draft_path, 'w') as f:\n",
|
||||
" f.write(scrape_result['markdown'])\n",
|
||||
" return self.state\n",
|
||||
"\n",
|
||||
" @router(scrape_blog_post)\n",
|
||||
" def select_platform(self):\n",
|
||||
" if self.state.post_type == \"twitter\":\n",
|
||||
" return \"twitter\"\n",
|
||||
" elif self.state.post_type == \"linkedin\":\n",
|
||||
" return \"linkedin\"\n",
|
||||
"\n",
|
||||
" @listen(\"twitter\")\n",
|
||||
" def twitter_draft(self):\n",
|
||||
" print(f\"# Planning content for: {self.state.draft_path}\")\n",
|
||||
" result = planning_crew.kickoff(inputs={'draft_path': self.state.draft_path, 'path_to_example_threads': self.state.path_to_example_threads})\n",
|
||||
" print(f\"# Planned content for {self.state.draft_path}:\")\n",
|
||||
" for tweet in result.pydantic.tweets:\n",
|
||||
" print(f\" - {tweet.content}\")\n",
|
||||
" return result\n",
|
||||
" \n",
|
||||
" @listen(\"linkedin\")\n",
|
||||
" def linkedin_draft(self):\n",
|
||||
" print(f\"# Planning content for: {self.state.draft_path}\")\n",
|
||||
" result = linkedin_planning_crew.kickoff(inputs={'draft_path': self.state.draft_path})\n",
|
||||
" print(f\"# Planned content for {self.state.draft_path}:\")\n",
|
||||
" print(f\" - {result.pydantic.content}\")\n",
|
||||
" return result\n",
|
||||
"\n",
|
||||
" @listen(or_(twitter_draft, linkedin_draft))\n",
|
||||
" def save_plan(self, plan):\n",
|
||||
" with open(f'thread/{self.state.draft_path.split(\"/\")[-1]}_{self.state.post_type}.json', 'w') as f:\n",
|
||||
" json.dump(plan.pydantic.model_dump(), f, indent=2)\n",
|
||||
"\n",
|
||||
" @listen(or_(twitter_draft, linkedin_draft))\n",
|
||||
" def publish(self, plan):\n",
|
||||
" print(f\"# Publishing thread for: {self.state.draft_path}\")\n",
|
||||
" ## Schedule for 1 hour from now \n",
|
||||
" response = scheduler.schedule(\n",
|
||||
" thread_model=plan,\n",
|
||||
" post_type=self.state.post_type\n",
|
||||
" )\n",
|
||||
" print(f\"# Thread scheduled for: {self.state.draft_path}\")\n",
|
||||
" print(f\"Here's the link to scheduled draft: {response['share_url']}\")\n",
|
||||
"\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Implementing helper methods to plot and execute the flow in a Jupyter notebook"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 43,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Plot saved as crewai_flow.html\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"# Plot the flow\n",
|
||||
"flow = CreateContentPlanningFlow()\n",
|
||||
"flow.plot()\n",
|
||||
"\n",
|
||||
"# Display the flow visualization using IFrame\n",
|
||||
"from IPython.display import IFrame\n",
|
||||
"\n",
|
||||
"# Display the flow visualization\n",
|
||||
"# IFrame(src='./crewai_flow.html', width='100%', height=400)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 44,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"ContentPlanningState(blog_post_url='https://blog.dailydoseofds.com/p/5-chunking-strategies-for-rag', draft_path='workdir/', post_type='twitter', path_to_example_threads='workdir/example_threads.txt')"
|
||||
]
|
||||
},
|
||||
"execution_count": 44,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"post_type = \"twitter\"\n",
|
||||
"flow = CreateContentPlanningFlow()\n",
|
||||
"flow.state.post_type = post_type\n",
|
||||
"flow.state"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 14,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"# fetching draft from: https://blog.dailydoseofds.com/p/5-chunking-strategies-for-rag\n",
|
||||
"# Planning content for: workdir/5 Chunking Strategies For RAG - by Avi Chawla.md\n",
|
||||
"# Planned content for workdir/5 Chunking Strategies For RAG - by Avi Chawla.md:\n",
|
||||
" - 5 Chunking Strategies For RAG\n",
|
||||
" - What is Retrieval-Augmented Generation (RAG)? 🤔 It's a technique that boosts language models with information retrieval, refining the interaction with large data. Chunking plays a crucial role here!\n",
|
||||
" - 1️⃣ Fixed-size Chunking: This method splits text into uniform segments. While straightforward, it risks losing contextual meaning. Use with care! 🔍\n",
|
||||
" - 2️⃣ Semantic Chunking: Here, we create chunks based on meaningful units (like sentences). This ensures better coherence and relevance in information retrieval. 🤝\n",
|
||||
" - 3️⃣ Recursive Chunking: Breaks text using natural language boundaries, like paragraphs, and refines chunks that are too large. Preserve flow while keeping it efficient! 📏\n",
|
||||
" - 4️⃣ Document Structure-based Chunking: Leverage document headings and sections for chunk boundaries. This maintains logical structure and aids retrieval accuracy! 📚\n",
|
||||
" - 5️⃣ LLM-based Chunking: Use large language models to generate semantically-rich chunks. This is powerful but comes with high computational costs. ⚡️\n",
|
||||
" - Key Takeaway: Each chunking strategy has its own strengths and weaknesses. Select the right method based on your model and content to optimize performance! 🚀\n",
|
||||
" - If you're diving into RAG, embrace chunking. Explore these strategies and elevate your AI's ability to communicate knowledge effectively! Join the discussion! 💬\n",
|
||||
"# Publishing thread for: workdir/5 Chunking Strategies For RAG - by Avi Chawla.md\n",
|
||||
"######## Thread JSON: {'topic': '5 Chunking Strategies For RAG', 'tweets': [{'content': '5 Chunking Strategies For RAG', 'is_hook': True, 'media_urls': []}, {'content': \"What is Retrieval-Augmented Generation (RAG)? 🤔 It's a technique that boosts language models with information retrieval, refining the interaction with large data. Chunking plays a crucial role here!\", 'is_hook': False, 'media_urls': ['https://substackcdn.com/image/fetch/w_1456,c_limit,f_auto,q_auto:good,fl_lossy/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6878b8fa-5e74-45a1-9a89-5aab92889126_2366x990.gif']}, {'content': '1️⃣ Fixed-size Chunking: This method splits text into uniform segments. While straightforward, it risks losing contextual meaning. Use with care! 🔍', 'is_hook': False, 'media_urls': ['https://substackcdn.com/image/fetch/w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F98c422a0-f0e2-457c-a256-4476a56a601f_943x232.png']}, {'content': '2️⃣ Semantic Chunking: Here, we create chunks based on meaningful units (like sentences). This ensures better coherence and relevance in information retrieval. 🤝', 'is_hook': False, 'media_urls': ['https://substackcdn.com/image/fetch/w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa6ad83a6-2879-4c77-9e49-393f16577aef_1066x288.gif']}, {'content': '3️⃣ Recursive Chunking: Breaks text using natural language boundaries, like paragraphs, and refines chunks that are too large. Preserve flow while keeping it efficient! 📏', 'is_hook': False, 'media_urls': ['https://substackcdn.com/image/fetch/w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff4009caa-34fc-48d6-8102-3d0f6f2c1386_1066x316.gif']}, {'content': '4️⃣ Document Structure-based Chunking: Leverage document headings and sections for chunk boundaries. This maintains logical structure and aids retrieval accuracy! 📚', 'is_hook': False, 'media_urls': ['https://substackcdn.com/image/fetch/w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe8febecd-ee68-42ff-ab06-41a0a3a43cd3_1102x306.gif']}, {'content': '5️⃣ LLM-based Chunking: Use large language models to generate semantically-rich chunks. This is powerful but comes with high computational costs. ⚡️', 'is_hook': False, 'media_urls': ['https://substackcdn.com/image/fetch/w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4d1b6d60-8956-4030-8525-d899ee61a9d5_1140x198.gif']}, {'content': 'Key Takeaway: Each chunking strategy has its own strengths and weaknesses. Select the right method based on your model and content to optimize performance! 🚀', 'is_hook': False, 'media_urls': []}, {'content': \"If you're diving into RAG, embrace chunking. Explore these strategies and elevate your AI's ability to communicate knowledge effectively! Join the discussion! 💬\", 'is_hook': False, 'media_urls': []}]}\n",
|
||||
"Thread scheduled successfully!\n",
|
||||
"# Thread scheduled for: workdir/5 Chunking Strategies For RAG - by Avi Chawla.md\n",
|
||||
"Here's the link to scheduled draft: https://typefully.com/t/hXG9c1d\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"\n",
|
||||
"flow.kickoff()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 39,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"flow = CreateContentPlanningFlow()\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": []
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 42,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"ContentPlanningState(blog_post_url='https://blog.dailydoseofds.com/p/5-chunking-strategies-for-rag', draft_path='workdir/', post_type='linkedin')"
|
||||
]
|
||||
},
|
||||
"execution_count": 42,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"flow.state"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": []
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.10.15"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 2
|
||||
}
|
||||
@@ -0,0 +1,3 @@
|
||||
crewai[tools]==0.86.0
|
||||
tqdm==4.67.1
|
||||
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 738 KiB |
@@ -0,0 +1,112 @@
|
||||
import os
|
||||
import json
|
||||
import requests
|
||||
import datetime
|
||||
from typing import Optional, Dict, Any
|
||||
from pydantic import BaseModel
|
||||
|
||||
# Constants
|
||||
API_URL = "https://api.typefully.com/v1/drafts/"
|
||||
API_KEY = os.getenv("TYPEFULLY_API_KEY")
|
||||
HEADERS = {
|
||||
"X-API-KEY": f"Bearer {API_KEY}"
|
||||
}
|
||||
|
||||
def json_to_typefully_content(thread_json: Dict[str, Any]) -> str:
|
||||
"""Convert JSON thread format to Typefully's format with 4 newlines between tweets."""
|
||||
tweets = thread_json['tweets']
|
||||
formatted_tweets = []
|
||||
for tweet in tweets:
|
||||
tweet_text = tweet['content']
|
||||
if 'media_urls' in tweet and tweet['media_urls']:
|
||||
tweet_text += f"\n{tweet['media_urls'][0]}"
|
||||
formatted_tweets.append(tweet_text)
|
||||
|
||||
return '\n\n\n\n'.join(formatted_tweets)
|
||||
|
||||
def json_to_linkedin_content(thread_json: Dict[str, Any]) -> str:
|
||||
"""Convert JSON thread format to Typefully's format."""
|
||||
content = thread_json['content']
|
||||
if 'url' in thread_json and thread_json['url']:
|
||||
content += f"\n{thread_json['url']}"
|
||||
return content
|
||||
|
||||
def schedule_thread(
|
||||
content: str,
|
||||
schedule_date: str = "next-free-slot",
|
||||
threadify: bool = False,
|
||||
share: bool = False,
|
||||
auto_retweet_enabled: bool = False,
|
||||
auto_plug_enabled: bool = False
|
||||
) -> Optional[Dict[str, Any]]:
|
||||
"""Schedule a thread on Typefully."""
|
||||
payload = {
|
||||
"content": content,
|
||||
"schedule-date": schedule_date,
|
||||
"threadify": threadify,
|
||||
"share": share,
|
||||
"auto_retweet_enabled": auto_retweet_enabled,
|
||||
"auto_plug_enabled": auto_plug_enabled
|
||||
}
|
||||
|
||||
payload = {key: value for key, value in payload.items() if value is not None}
|
||||
|
||||
try:
|
||||
response = requests.post(API_URL, json=payload, headers=HEADERS)
|
||||
response.raise_for_status()
|
||||
return response.json()
|
||||
except requests.exceptions.RequestException as e:
|
||||
print(f"Error: {e}")
|
||||
return None
|
||||
|
||||
def schedule(
|
||||
thread_model: BaseModel,
|
||||
hours_from_now: int = 1,
|
||||
threadify: bool = False,
|
||||
share: bool = True,
|
||||
post_type: str = "twitter"
|
||||
) -> Optional[Dict[str, Any]]:
|
||||
"""
|
||||
Schedule a thread from a Pydantic model.
|
||||
|
||||
Args:
|
||||
thread_model: Pydantic model containing thread data
|
||||
hours_from_now: Hours from now to schedule the thread (default: 1)
|
||||
threadify: Whether to let Typefully split the content (default: False)
|
||||
share: Whether to get a share URL in response (default: True)
|
||||
|
||||
Returns:
|
||||
API response dictionary or None if failed
|
||||
"""
|
||||
try:
|
||||
# Convert Pydantic model to dict
|
||||
thread_json = thread_model.pydantic.model_dump()
|
||||
print("######## Thread JSON: ", thread_json)
|
||||
# Convert to Typefully format
|
||||
if post_type == "twitter":
|
||||
thread_content = json_to_typefully_content(thread_json)
|
||||
elif post_type == "linkedin":
|
||||
thread_content = json_to_linkedin_content(thread_json)
|
||||
|
||||
# Calculate schedule time
|
||||
schedule_date = (datetime.datetime.utcnow() +
|
||||
datetime.timedelta(hours=hours_from_now)).isoformat() + "Z"
|
||||
|
||||
# Schedule the thread
|
||||
response = schedule_thread(
|
||||
content=thread_content,
|
||||
schedule_date=schedule_date,
|
||||
threadify=threadify,
|
||||
share=share
|
||||
)
|
||||
|
||||
if response:
|
||||
print("Thread scheduled successfully!")
|
||||
return response
|
||||
else:
|
||||
print("Failed to schedule the thread.")
|
||||
return None
|
||||
|
||||
except Exception as e:
|
||||
print(f"Error: {str(e)}")
|
||||
return None
|
||||
Reference in New Issue
Block a user