{
"cells": [
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "9_CQrFgOj78b"
},
"outputs": [],
"source": [
"%%capture\n",
"!pip install scrapegraphai\n",
"!apt install chromium-chromedriver\n",
"!pip install nest_asyncio\n",
"!pip install playwright\n",
"!playwright install"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "tb33AcRHywFb"
},
"outputs": [],
"source": [
"import nest_asyncio\n",
"\n",
"nest_asyncio.apply()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "00a84YVhhxJr"
},
"outputs": [],
"source": [
"# correct APIKEY\n",
"OPENAI_API_KEY = \"YOUR API KEY\""
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "vGDjka17pqqg"
},
"source": [
"For more examples visit [the examples folder](https://github.com/ScrapeGraphAI/Scrapegraph-ai/tree/main/examples)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "Mrujgp-nlp12"
},
"source": [
"# SmartScraperGraph\n",
"**SmartScraperGraph** is a class representing one of the default scraping pipelines. It uses a direct graph implementation where each node has its own function, from retrieving html from a website to extracting relevant information based on your query and generate a coherent answer."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "M-dmSB0_zHCQ"
},
"source": [
""
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "uqYBNOM2YZD9"
},
"source": [
"## Using OpenAI models"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "ogiF4g5Z-bzG"
},
"outputs": [],
"source": [
"from scrapegraphai.graphs import SmartScraperGraph"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "7ZzONlJ6-oe_"
},
"source": [
"Define the configuration for the graph"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "MPZgrZ12-eRc"
},
"outputs": [],
"source": [
"graph_config = {\n",
" \"llm\": {\n",
" \"api_key\": OPENAI_API_KEY,\n",
" \"model\": \"openai/gpt-4o-mini\",\n",
" \"temperature\": 0,\n",
" },\n",
" \"verbose\": True,\n",
"}"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "DjDt_10r-q8P"
},
"source": [
"Create the SmartScraperGraph instance and run it"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "aV4VTnx9-h_d"
},
"outputs": [],
"source": [
"smart_scraper_graph = SmartScraperGraph(\n",
" prompt=\"List me all the projects with their descriptions.\",\n",
" # also accepts a string with the already downloaded HTML code\n",
" source=\"https://perinim.github.io/projects/\",\n",
" config=graph_config,\n",
")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "E3pyGQZLTiZ8"
},
"outputs": [],
"source": [
"graph_config = {\n",
" \"llm\": {\n",
" \"api_key\": OPENAI_API_KEY,\n",
" \"model\": \"openai/gpt-4o-mini\",\n",
" },\n",
" \"verbose\": True,\n",
" \"headless\": True,\n",
"}\n",
"\n",
"# ************************************************\n",
"# Create the SmartScraperGraph instance and run it\n",
"# ************************************************\n",
"\n",
"smart_scraper_graph = SmartScraperGraph(\n",
" prompt=\"List me all the projects with their description\",\n",
" source=\"https://perinim.github.io/projects/\",\n",
" config=graph_config,\n",
")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "Zty23idsAtwU",
"outputId": "419dd75f-18c6-44d2-da82-ca8967d17e0f"
},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"--- Executing Fetch Node ---\n",
"--- (Fetching HTML from: https://perinim.github.io/projects/) ---\n",
"--- Executing ParseNode Node ---\n",
"--- Executing GenerateAnswer Node ---\n"
]
}
],
"source": [
"result = smart_scraper_graph.run()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "rnGhLGCuAqRU",
"outputId": "062aeab2-3e96-4fec-d04a-b9acae142f40"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"{\n",
" \"projects\": [\n",
" {\n",
" \"name\": \"Rotary Pendulum RL\",\n",
" \"description\": \"Open Source project aimed at controlling a real life rotary pendulum using RL algorithms\"\n",
" },\n",
" {\n",
" \"name\": \"DQN Implementation from scratch\",\n",
" \"description\": \"Developed a Deep Q-Network algorithm to train a simple and double pendulum\"\n",
" },\n",
" {\n",
" \"name\": \"Multi Agents HAED\",\n",
" \"description\": \"University project which focuses on simulating a multi-agent system to perform environment mapping. Agents, equipped with sensors, explore and record their surroundings, considering uncertainties in their readings.\"\n",
" },\n",
" {\n",
" \"name\": \"Wireless ESC for Modular Drones\",\n",
" \"description\": \"Modular drone architecture proposal and proof of concept. The project received maximum grade.\"\n",
" }\n",
" ]\n",
"}\n"
]
}
],
"source": [
"import json\n",
"\n",
"output = json.dumps(result, indent=2)\n",
"\n",
"line_list = output.split(\"\\n\") # Sort of line replacing \"\\n\" with a new line\n",
"\n",
"for line in line_list:\n",
" print(line)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "5poLHYLVa-6E"
},
"source": [
"# Search graph\n",
"This graph **transforms** the user prompt in a **internet search query**, fetch the relevant URLs, and start the scraping process. Similar to the **SmartScraperGraph** but with the addition of the **SearchInternetNode** node."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "NRIoaXSzzP8M"
},
"source": [
""
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "RIvbQjyhbHhW"
},
"outputs": [],
"source": [
"from scrapegraphai.graphs import SearchGraph\n",
"\n",
"# Define the configuration for the graph\n",
"graph_config = {\n",
" \"llm\": {\n",
" \"api_key\": OPENAI_API_KEY,\n",
" \"model\": \"openai/gpt-4o-mini\",\n",
" \"temperature\": 0,\n",
" },\n",
"}\n",
"\n",
"# Create the SearchGraph instance\n",
"search_graph = SearchGraph(\n",
" prompt=\"List me all the European countries. Look in wikipedia.\", config=graph_config\n",
")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "XnVtc7SzCkUY"
},
"outputs": [],
"source": [
"result = search_graph.run()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "3LPAh-yQCqkY"
},
"source": [
"Prettify the result and display the JSON"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "xgnWDLTjzHwv",
"outputId": "f0c8ebf4-5ba5-4330-dbd8-1c9fdd93eaeb"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"{\n",
" \"European_countries\": [\n",
" \"Albania\",\n",
" \"Andorra\",\n",
" \"Armenia\",\n",
" \"Austria\",\n",
" \"Azerbaijan\",\n",
" \"Belarus\",\n",
" \"Belgium\",\n",
" \"Bosnia and Herzegovina\",\n",
" \"Bulgaria\",\n",
" \"Croatia\",\n",
" \"Cyprus\",\n",
" \"Czech Republic\",\n",
" \"Denmark\",\n",
" \"Estonia\",\n",
" \"Finland\",\n",
" \"France\",\n",
" \"Georgia\",\n",
" \"Germany\",\n",
" \"Greece\",\n",
" \"Hungary\",\n",
" \"Iceland\",\n",
" \"Ireland\",\n",
" \"Italy\",\n",
" \"Jersey\",\n",
" \"Isle of Man\",\n",
" \"Kazakhstan\",\n",
" \"Latvia\",\n",
" \"Liechtenstein\",\n",
" \"Lithuania\",\n",
" \"Luxembourg\",\n",
" \"Malta\",\n",
" \"Moldova\",\n",
" \"Monaco\",\n",
" \"Montenegro\",\n",
" \"Netherlands\",\n",
" \"North Macedonia\",\n",
" \"Norway\",\n",
" \"Poland\",\n",
" \"Portugal\",\n",
" \"Romania\",\n",
" \"Russia\",\n",
" \"San Marino\",\n",
" \"Serbia\",\n",
" \"Slovakia\",\n",
" \"Slovenia\",\n",
" \"Spain\",\n",
" \"Sweden\",\n",
" \"Switzerland\",\n",
" \"Turkey\",\n",
" \"Ukraine\",\n",
" \"United Kingdom\",\n",
" \"Vatican City\",\n",
" \"Kosovo\",\n",
" \"Gibraltar\",\n",
" \"Faroe Islands\",\n",
" \"Guernsey\",\n",
" \"Jersey\"\n",
" ],\n",
" \"sources\": [\n",
" \"https://simple.wikipedia.org/wiki/List_of_European_countries\",\n",
" \"https://en.wikipedia.org/wiki/List_of_European_countries_by_population\",\n",
" \"https://en.wikipedia.org/wiki/Member_state_of_the_European_Union\"\n",
" ]\n",
"}\n"
]
}
],
"source": [
"import json\n",
"\n",
"output = json.dumps(result, indent=2)\n",
"\n",
"line_list = output.split(\"\\n\") # Sort of line replacing \"\\n\" with a new line\n",
"\n",
"for line in line_list:\n",
" print(line)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "N5IMdKHvlXFY"
},
"source": [
"# SpeechGraph\n",
"**SpeechGraph** is a class representing one of the default scraping pipelines that generate the answer together with an audio file. Similar to the **SmartScraperGraph** but with the addition of the **TextToSpeechNode** node.\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "pqJsEVgizs-M"
},
"source": [
""
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "W9KhWlT3lXFd"
},
"outputs": [],
"source": [
"from scrapegraphai.graphs import SpeechGraph\n",
"\n",
"# Define the configuration for the graph\n",
"graph_config = {\n",
" \"llm\": {\n",
" \"api_key\": OPENAI_API_KEY,\n",
" \"model\": \"gpt-3.5-turbo\",\n",
" },\n",
" \"tts_model\": {\"api_key\": OPENAI_API_KEY, \"model\": \"tts-1\", \"voice\": \"alloy\"},\n",
" \"output_path\": \"website_summary.mp3\",\n",
"}\n",
"\n",
"# Create the SpeechGraph instance\n",
"speech_graph = SpeechGraph(\n",
" prompt=\"Create a summary of the website\",\n",
" source=\"https://perinim.github.io/projects/\",\n",
" config=graph_config,\n",
")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "nVolb3paEczD",
"outputId": "d7d316a0-7580-4a6c-8f20-7e1cb1fc3f07"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"--- Executing Fetch Node ---\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"Fetching pages: 100%|##########| 1/1 [00:00<00:00, 17.07it/s]\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"--- Executing Parse Node ---\n",
"--- Executing RAG Node ---\n",
"--- (updated chunks metadata) ---\n",
"--- (tokens compressed and vector stored) ---\n",
"--- Executing GenerateAnswer Node ---\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"Processing chunks: 100%|██████████| 1/1 [00:00<00:00, 339.78it/s]\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"--- Executing TextToSpeech Node ---\n",
"Audio saved to website_summary.mp3\n"
]
}
],
"source": [
"result = speech_graph.run()\n",
"answer = result.get(\"answer\", \"No answer found\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "znt2EOKZE3z2"
},
"source": [
"Prettify the result and display the JSON"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "QqY0TbwbEp-O",
"outputId": "c2b1127d-0c49-4121-922e-39da65c329ee"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"{\n",
" \"summary\": {\n",
" \"title\": \"Projects | \",\n",
" \"projects\": [\n",
" {\n",
" \"title\": \"Rotary Pendulum RL\",\n",
" \"description\": \"Open Source project aimed at controlling a real life rotary pendulum using RL algorithms\"\n",
" },\n",
" {\n",
" \"title\": \"DQN Implementation from scratch\",\n",
" \"description\": \"Developed a Deep Q-Network algorithm to train a simple and double pendulum\"\n",
" },\n",
" {\n",
" \"title\": \"Multi Agents HAED\",\n",
" \"description\": \"University project which focuses on simulating a multi-agent system to perform environment mapping. Agents, equipped with sensors, explore and record their surroundings, considering uncertainties in their readings.\"\n",
" },\n",
" {\n",
" \"title\": \"Wireless ESC for Modular Drones\",\n",
" \"description\": \"Modular drone architecture proposal and proof of concept. The project received maximum grade.\"\n",
" }\n",
" ]\n",
" }\n",
"}\n"
]
}
],
"source": [
"import json\n",
"\n",
"output = json.dumps(answer, indent=2)\n",
"\n",
"line_list = output.split(\"\\n\") # Sort of line replacing \"\\n\" with a new line\n",
"\n",
"for line in line_list:\n",
" print(line)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 75
},
"id": "lfJ_jVwklXFd",
"outputId": "dc4ad491-4422-4edb-91ae-35775b23168a"
},
"outputs": [
{
"data": {
"text/html": [
"\n",
" \n",
" "
],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"from IPython.display import Audio\n",
"\n",
"wn = Audio(\"website_summary.mp3\", autoplay=True)\n",
"display(wn)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "p9kC0x4NuLTx"
},
"source": [
"# Build a Custom Graph\n",
"It is possible to **build your own scraping pipeline** by using the default nodes and place them as you wish, without using pre-defined graphs."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "Pr6DIqt2uLUI"
},
"source": [
"You can create **custom graphs** based on your necessities, using standard nodes provided by the library.\n",
"\n",
"The list of the existing nodes can be found through the *nodes_metadata* json construct.\n",
"\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "-o29vDSIvG4t",
"outputId": "be469b65-ba01-437a-e217-ed1c4f3ad264"
},
"outputs": [
{
"data": {
"text/plain": [
"dict_keys(['SearchInternetNode', 'FetchNode', 'GetProbableTagsNode', 'ParseNode', 'RAGNode', 'GenerateAnswerNode', 'ConditionalNode', 'ImageToTextNode', 'TextToSpeechNode'])"
]
},
"execution_count": 17,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# check available nodes\n",
"from scrapegraphai.helpers import nodes_metadata\n",
"\n",
"nodes_metadata.keys()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "829wW5E6vrjJ",
"outputId": "58203025-64ce-4107-f6d3-3b3cfa5537d5"
},
"outputs": [
{
"data": {
"text/plain": [
"{'description': 'Converts image content to text by \\n extracting visual information and interpreting it.',\n",
" 'type': 'node',\n",
" 'args': {'image_data': 'Data of the image to be processed.'},\n",
" 'returns': \"Updated state with the textual description of the image under 'image_text' key.\"}"
]
},
"execution_count": 18,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# to get more information about a node\n",
"nodes_metadata[\"ImageToTextNode\"]"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "3pnNFDckwWy7"
},
"source": [
"To create a custom graph we must:\n",
"\n",
"1. **Istantiate the nodes** you want to use\n",
"2. Create the graph using **BaseGraph** class, which must have a **list of nodes**, tuples representing the **edges** of the graph, an **entry_point**\n",
"3. Run it using the **execute** method\n",
"\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "eQLZJyg4uLUJ"
},
"outputs": [],
"source": [
"from langchain_openai import OpenAIEmbeddings\n",
"from scrapegraphai.models import OpenAI\n",
"from scrapegraphai.graphs import BaseGraph\n",
"from scrapegraphai.nodes import FetchNode, ParseNode, RAGNode, GenerateAnswerNode\n",
"\n",
"# Define the configuration for the graph\n",
"graph_config = {\n",
" \"llm\": {\n",
" \"api_key\": OPENAI_API_KEY,\n",
" \"model\": \"openai/gpt-4o\",\n",
" \"temperature\": 0,\n",
" \"streaming\": True,\n",
" },\n",
"}\n",
"\n",
"llm_model = OpenAI(graph_config[\"llm\"])\n",
"embedder = OpenAIEmbeddings(api_key=llm_model.openai_api_key)\n",
"\n",
"# define the nodes for the graph\n",
"fetch_node = FetchNode(\n",
" input=\"url | local_dir\",\n",
" output=[\"doc\", \"link_urls\", \"img_urls\"],\n",
" node_config={\n",
" \"verbose\": True,\n",
" \"headless\": True,\n",
" },\n",
")\n",
"parse_node = ParseNode(\n",
" input=\"doc\",\n",
" output=[\"parsed_doc\"],\n",
" node_config={\n",
" \"chunk_size\": 4096,\n",
" \"verbose\": True,\n",
" },\n",
")\n",
"rag_node = RAGNode(\n",
" input=\"user_prompt & (parsed_doc | doc)\",\n",
" output=[\"relevant_chunks\"],\n",
" node_config={\n",
" \"llm_model\": llm_model,\n",
" \"embedder_model\": embedder,\n",
" \"verbose\": True,\n",
" },\n",
")\n",
"generate_answer_node = GenerateAnswerNode(\n",
" input=\"user_prompt & (relevant_chunks | parsed_doc | doc)\",\n",
" output=[\"answer\"],\n",
" node_config={\n",
" \"llm_model\": llm_model,\n",
" \"verbose\": True,\n",
" },\n",
")\n",
"\n",
"# create the graph by defining the nodes and their connections\n",
"graph = BaseGraph(\n",
" nodes=[\n",
" fetch_node,\n",
" parse_node,\n",
" rag_node,\n",
" generate_answer_node,\n",
" ],\n",
" edges=[\n",
" (fetch_node, parse_node),\n",
" (parse_node, rag_node),\n",
" (rag_node, generate_answer_node),\n",
" ],\n",
" entry_point=fetch_node,\n",
")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "5FYKF9H1Fvb8",
"outputId": "666d51fe-5e2f-4398-a3b0-bb820960a0d1"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"--- Executing Fetch Node ---\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"Fetching pages: 100%|##########| 1/1 [00:00<00:00, 28.65it/s]\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"--- Executing Parse Node ---\n",
"--- Executing RAG Node ---\n",
"--- (updated chunks metadata) ---\n",
"--- (tokens compressed and vector stored) ---\n",
"--- Executing GenerateAnswer Node ---\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"Processing chunks: 100%|██████████| 1/1 [00:00<00:00, 911.01it/s]\n"
]
}
],
"source": [
"# execute the graph\n",
"result, execution_info = graph.execute(\n",
" {\n",
" \"user_prompt\": \"List me the projects with their description\",\n",
" \"url\": \"https://perinim.github.io/projects/\",\n",
" }\n",
")\n",
"\n",
"# get the answer from the result\n",
"result = result.get(\"answer\", \"No answer found.\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "JEP8_zZ9GHW2"
},
"source": [
"Prettify the result and display the JSON"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "nx9qGaxvFmfT",
"outputId": "fb327a6a-0dfa-417b-8dbb-505bebc96fe8"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"{\n",
" \"projects\": [\n",
" {\n",
" \"title\": \"Rotary Pendulum RL\",\n",
" \"description\": \"Open Source project aimed at controlling a real life rotary pendulum using RL algorithms\"\n",
" },\n",
" {\n",
" \"title\": \"DQN Implementation from scratch\",\n",
" \"description\": \"Developed a Deep Q-Network algorithm to train a simple and double pendulum\"\n",
" },\n",
" {\n",
" \"title\": \"Multi Agents HAED\",\n",
" \"description\": \"University project which focuses on simulating a multi-agent system to perform environment mapping. Agents, equipped with sensors, explore and record their surroundings, considering uncertainties in their readings.\"\n",
" },\n",
" {\n",
" \"title\": \"Wireless ESC for Modular Drones\",\n",
" \"description\": \"Modular drone architecture proposal and proof of concept. The project received maximum grade.\"\n",
" }\n",
" ]\n",
"}\n"
]
}
],
"source": [
"import json\n",
"\n",
"output = json.dumps(result, indent=2)\n",
"\n",
"line_list = output.split(\"\\n\") # Sort of line replacing \"\\n\" with a new line\n",
"\n",
"for line in line_list:\n",
" print(line)"
]
}
],
"metadata": {
"colab": {
"collapsed_sections": [
"N5IMdKHvlXFY"
],
"provenance": []
},
"kernelspec": {
"display_name": "Python 3",
"name": "python3"
},
"language_info": {
"name": "python"
}
},
"nbformat": 4,
"nbformat_minor": 0
}