wehub-resource-sync
6b7e6b44f1
Python Build and Type Check / python-ci (ubuntu-latest, 3.11) (push) Has been cancelled
Python Build and Type Check / python-ci (ubuntu-latest, 3.13) (push) Has been cancelled
Python Build and Type Check / python-ci (windows-latest, 3.11) (push) Has been cancelled
Python Build and Type Check / python-ci (windows-latest, 3.13) (push) Has been cancelled
Python Integration Tests / python-ci (ubuntu-latest, 3.13) (push) Has been cancelled
Python Integration Tests / python-ci (windows-latest, 3.13) (push) Has been cancelled
Python Notebook Tests / python-ci (ubuntu-latest, 3.13) (push) Has been cancelled
Python Notebook Tests / python-ci (windows-latest, 3.13) (push) Has been cancelled
Python Smoke Tests / python-ci (ubuntu-latest, 3.13) (push) Has been cancelled
Python Smoke Tests / python-ci (windows-latest, 3.13) (push) Has been cancelled
Python Unit Tests / python-ci (ubuntu-latest, 3.13) (push) Has been cancelled
Python Unit Tests / python-ci (windows-latest, 3.13) (push) Has been cancelled
gh-pages / build (push) Has been cancelled
Python Publish (pypi) / Upload release to PyPI (push) Has been cancelled
Spellcheck / spellcheck (push) Has been cancelled
537 lines
19 KiB
Plaintext
537 lines
19 KiB
Plaintext
{
|
|
"cells": [
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "91a0ee2b",
|
|
"metadata": {},
|
|
"source": [
|
|
"# Batching\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "422fcc73",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Completion Batching\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "88e715fe",
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"In the velvet silence of the night,\n",
|
|
"A canvas vast and infinite unfolds,\n",
|
|
"Where stories of the cosmos,\n",
|
|
"Metrics for: azure/gpt-4o\n",
|
|
"{\n",
|
|
" \"attempted_request_count\": 10,\n",
|
|
" \"successful_response_count\": 10,\n",
|
|
" \"failed_response_count\": 0,\n",
|
|
" \"failure_rate\": 0.0,\n",
|
|
" \"requests_with_retries\": 0,\n",
|
|
" \"retries\": 0,\n",
|
|
" \"retry_rate\": 0.0,\n",
|
|
" \"compute_duration_seconds\": 157.70280289649963,\n",
|
|
" \"compute_duration_per_response_seconds\": 15.770280289649964,\n",
|
|
" \"runtime_duration_seconds\": 18.660003900527954,\n",
|
|
" \"streaming_responses\": 0,\n",
|
|
" \"responses_with_tokens\": 10,\n",
|
|
" \"prompt_tokens\": 280,\n",
|
|
" \"completion_tokens\": 9234,\n",
|
|
" \"total_tokens\": 9514,\n",
|
|
" \"tokens_per_response\": 951.4,\n",
|
|
" \"responses_with_cost\": 10,\n",
|
|
" \"input_cost\": 0.0007,\n",
|
|
" \"output_cost\": 0.09234,\n",
|
|
" \"total_cost\": 0.09304000000000001,\n",
|
|
" \"cost_per_response\": 0.009304000000000002\n",
|
|
"}\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"# Copyright (c) 2024 Microsoft Corporation.\n",
|
|
"# Licensed under the MIT License\n",
|
|
"\n",
|
|
"import json\n",
|
|
"import os\n",
|
|
"\n",
|
|
"from dotenv import load_dotenv\n",
|
|
"from graphrag_llm.completion import LLMCompletion, create_completion\n",
|
|
"from graphrag_llm.config import AuthMethod, ModelConfig\n",
|
|
"from graphrag_llm.types import LLMCompletionArgs\n",
|
|
"\n",
|
|
"load_dotenv()\n",
|
|
"\n",
|
|
"api_key = os.getenv(\"GRAPHRAG_API_KEY\")\n",
|
|
"model_config = ModelConfig(\n",
|
|
" model_provider=\"azure\",\n",
|
|
" model=os.getenv(\"GRAPHRAG_MODEL\", \"gpt-4o\"),\n",
|
|
" azure_deployment_name=os.getenv(\"GRAPHRAG_MODEL\", \"gpt-4o\"),\n",
|
|
" api_base=os.getenv(\"GRAPHRAG_API_BASE\"),\n",
|
|
" api_version=os.getenv(\"GRAPHRAG_API_VERSION\", \"2025-04-01-preview\"),\n",
|
|
" api_key=api_key,\n",
|
|
" auth_method=AuthMethod.AzureManagedIdentity if not api_key else AuthMethod.ApiKey,\n",
|
|
")\n",
|
|
"llm_completion: LLMCompletion = create_completion(model_config)\n",
|
|
"\n",
|
|
"\n",
|
|
"completion_requests: list[LLMCompletionArgs] = [\n",
|
|
" {\n",
|
|
" \"messages\": \"Write a 1000 word poem about the night sky and all the wonders and mysteries of the universe.\"\n",
|
|
" },\n",
|
|
"] * 10\n",
|
|
"\n",
|
|
"# Spins up to 25 concurrent requests\n",
|
|
"# Which is more than the number of requests being made\n",
|
|
"# and since rate limiting is not enabled, all the requests fire off immediately\n",
|
|
"# and complete as fast as the LLM provider allows\n",
|
|
"responses = llm_completion.completion_batch(completion_requests, concurrency=25)\n",
|
|
"for response in responses:\n",
|
|
" if isinstance(response, Exception):\n",
|
|
" print(f\"Error: {response}\")\n",
|
|
" else:\n",
|
|
" # Print the first 100 characters of the first successful response\n",
|
|
" print(response.content[0:100]) # type: ignore\n",
|
|
" break\n",
|
|
"\n",
|
|
"print(f\"Metrics for: {llm_completion.metrics_store.id}\")\n",
|
|
"print(json.dumps(llm_completion.metrics_store.get_metrics(), indent=2))"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "668e1f94",
|
|
"metadata": {},
|
|
"source": [
|
|
"Notice the difference between `compute_duration_seconds` and `runtime_duration_seconds`. The former indicates how long all the network requests took to complete and would be how long the whole process took to complete if running the requests in series. The latter indicates how long the batch as a whole took to complete when running with concurrency.\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "49ec7716",
|
|
"metadata": {},
|
|
"source": [
|
|
"### With Rate Limiting\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 2,
|
|
"id": "eb73f940",
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Metrics for: azure/gpt-4o\n",
|
|
"{\n",
|
|
" \"attempted_request_count\": 10,\n",
|
|
" \"successful_response_count\": 10,\n",
|
|
" \"failed_response_count\": 0,\n",
|
|
" \"failure_rate\": 0.0,\n",
|
|
" \"requests_with_retries\": 0,\n",
|
|
" \"retries\": 0,\n",
|
|
" \"retry_rate\": 0.0,\n",
|
|
" \"compute_duration_seconds\": 108.16670417785645,\n",
|
|
" \"compute_duration_per_response_seconds\": 10.816670417785645,\n",
|
|
" \"runtime_duration_seconds\": 38.489975929260254,\n",
|
|
" \"streaming_responses\": 0,\n",
|
|
" \"responses_with_tokens\": 10,\n",
|
|
" \"prompt_tokens\": 280,\n",
|
|
" \"completion_tokens\": 8965,\n",
|
|
" \"total_tokens\": 9245,\n",
|
|
" \"tokens_per_response\": 924.5,\n",
|
|
" \"responses_with_cost\": 10,\n",
|
|
" \"input_cost\": 0.0007,\n",
|
|
" \"output_cost\": 0.08965000000000002,\n",
|
|
" \"total_cost\": 0.09035000000000001,\n",
|
|
" \"cost_per_response\": 0.009035000000000001\n",
|
|
"}\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"from graphrag_llm.config import RateLimitConfig, RateLimitType\n",
|
|
"\n",
|
|
"model_config.rate_limit = RateLimitConfig(\n",
|
|
" type=RateLimitType.SlidingWindow,\n",
|
|
" period_in_seconds=60, # limit requests per minute\n",
|
|
" requests_per_period=20, # max 20 requests per minute. Fire one off every 3 seconds\n",
|
|
")\n",
|
|
"llm_completion: LLMCompletion = create_completion(model_config)\n",
|
|
"llm_completion.metrics_store.clear_metrics()\n",
|
|
"\n",
|
|
"responses = llm_completion.completion_batch(completion_requests, concurrency=25)\n",
|
|
"\n",
|
|
"print(f\"Metrics for: {llm_completion.metrics_store.id}\")\n",
|
|
"print(json.dumps(llm_completion.metrics_store.get_metrics(), indent=2))"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "ceb93f24",
|
|
"metadata": {},
|
|
"source": [
|
|
"Notice the `runtime_duration_seconds` is now much slower as the requests are being throttled by the rate limit.\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "05bd00e6",
|
|
"metadata": {},
|
|
"source": [
|
|
"### With Cache\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 4,
|
|
"id": "3cb345ec",
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Metrics for: azure/gpt-4o\n",
|
|
"{\n",
|
|
" \"attempted_request_count\": 10,\n",
|
|
" \"successful_response_count\": 10,\n",
|
|
" \"failed_response_count\": 0,\n",
|
|
" \"failure_rate\": 0.0,\n",
|
|
" \"requests_with_retries\": 0,\n",
|
|
" \"retries\": 0,\n",
|
|
" \"retry_rate\": 0.0,\n",
|
|
" \"compute_duration_seconds\": 93.54697012901306,\n",
|
|
" \"compute_duration_per_response_seconds\": 9.354697012901307,\n",
|
|
" \"runtime_duration_seconds\": 10.748144149780273,\n",
|
|
" \"cached_responses\": 6,\n",
|
|
" \"streaming_responses\": 0,\n",
|
|
" \"responses_with_tokens\": 10,\n",
|
|
" \"prompt_tokens\": 280,\n",
|
|
" \"completion_tokens\": 7869,\n",
|
|
" \"total_tokens\": 8149,\n",
|
|
" \"tokens_per_response\": 814.9,\n",
|
|
" \"responses_with_cost\": 10,\n",
|
|
" \"input_cost\": 0.0007,\n",
|
|
" \"output_cost\": 0.07869000000000001,\n",
|
|
" \"total_cost\": 0.07939000000000002,\n",
|
|
" \"cost_per_response\": 0.007939000000000002\n",
|
|
"}\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"from graphrag_cache import create_cache\n",
|
|
"\n",
|
|
"cache = create_cache()\n",
|
|
"\n",
|
|
"# Redisable rate limiting\n",
|
|
"model_config.rate_limit = None\n",
|
|
"\n",
|
|
"llm_completion: LLMCompletion = create_completion(model_config, cache=cache)\n",
|
|
"llm_completion.metrics_store.clear_metrics()\n",
|
|
"\n",
|
|
"responses = llm_completion.completion_batch(completion_requests, concurrency=4)\n",
|
|
"\n",
|
|
"print(f\"Metrics for: {llm_completion.metrics_store.id}\")\n",
|
|
"print(json.dumps(llm_completion.metrics_store.get_metrics(), indent=2))"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "6de2c4cf",
|
|
"metadata": {},
|
|
"source": [
|
|
"Notice the `cached_responses == 6` since we are spinning up `4` threads. The first 4 requests are fired off immediately prior to any data in the cache. This means when identical requests are fired in the same thread cycle they will all hit the model since the cache is not yet populated.\n",
|
|
"\n",
|
|
"The `cached_responses` indicates how many cache hits occurred but the rest of the metrics exist as if a cache was not used. For example, `compute_duration_seconds` and all the tokens and cost counts are as if cache was not used so `compute_duration_seconds` includes network timings for the cached responses. This is because both the response and metrics are cached and retrieved from the cache when a cache hit occurs. This means the above metrics should closely match the metrics from the first example in this notebook other than the `runtime_duration_seconds` which gives the true idea of how long a job takes to run. Rerunning a job with a fully hydrated cache should result in a quick `runtime_duration_seconds`. Metrics were designed to give an idea of how long and costly a job would be if there were no cache.\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "3e6d20d6",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Embedding Batching\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "2e95c4e6",
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Embedding vector length: 1536\n",
|
|
"[0.012382184155285358, -0.0487498939037323, 0.02962493523955345, 0.0321056991815567, -0.030259549617767334]\n",
|
|
"Embedding vector length: 1536\n",
|
|
"[-0.01842353865504265, -0.00725775770843029, 0.0036669441033154726, -0.0542047917842865, -0.022724902257323265]\n",
|
|
"Embedding vector length: 1536\n",
|
|
"[-0.055969491600990295, 0.023217301815748215, -0.007630861829966307, 0.002210293198004365, 0.01284848153591156]\n",
|
|
"Metrics for: azure/text-embedding-3-small\n",
|
|
"{\n",
|
|
" \"attempted_request_count\": 2,\n",
|
|
" \"successful_response_count\": 2,\n",
|
|
" \"failed_response_count\": 0,\n",
|
|
" \"failure_rate\": 0.0,\n",
|
|
" \"requests_with_retries\": 0,\n",
|
|
" \"retries\": 0,\n",
|
|
" \"retry_rate\": 0.0,\n",
|
|
" \"compute_duration_seconds\": 2.0372798442840576,\n",
|
|
" \"compute_duration_per_response_seconds\": 1.0186399221420288,\n",
|
|
" \"runtime_duration_seconds\": 1.02105712890625,\n",
|
|
" \"streaming_responses\": 0,\n",
|
|
" \"responses_with_tokens\": 2,\n",
|
|
" \"prompt_tokens\": 23,\n",
|
|
" \"total_tokens\": 23,\n",
|
|
" \"tokens_per_response\": 11.5,\n",
|
|
" \"responses_with_cost\": 2,\n",
|
|
" \"input_cost\": 4.6e-07,\n",
|
|
" \"total_cost\": 4.6e-07,\n",
|
|
" \"cost_per_response\": 2.3e-07\n",
|
|
"}\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"from graphrag_llm.embedding import LLMEmbedding, create_embedding\n",
|
|
"from graphrag_llm.types import LLMEmbeddingArgs\n",
|
|
"\n",
|
|
"embedding_config = ModelConfig(\n",
|
|
" model_provider=\"azure\",\n",
|
|
" model=os.getenv(\"GRAPHRAG_EMBEDDING_MODEL\", \"text-embedding-3-small\"),\n",
|
|
" azure_deployment_name=os.getenv(\n",
|
|
" \"GRAPHRAG_LLM_EMBEDDING_MODEL\", \"text-embedding-3-small\"\n",
|
|
" ),\n",
|
|
" api_base=os.getenv(\"GRAPHRAG_API_BASE\"),\n",
|
|
" api_version=os.getenv(\"GRAPHRAG_API_VERSION\", \"2025-04-01-preview\"),\n",
|
|
" api_key=api_key,\n",
|
|
" auth_method=AuthMethod.AzureManagedIdentity if not api_key else AuthMethod.ApiKey,\n",
|
|
")\n",
|
|
"\n",
|
|
"llm_embedding: LLMEmbedding = create_embedding(embedding_config)\n",
|
|
"\n",
|
|
"# A single embedding request already accepts a list of inputs to embed\n",
|
|
"# Here we demonstrate batching multiple embedding requests concurrently\n",
|
|
"# The first request has two inputs to embed and the second has one input\n",
|
|
"embedding_requests: list[LLMEmbeddingArgs] = [\n",
|
|
" {\"input\": [\"Hello World.\", \"The quick brown fox jumps over the lazy dog.\"]},\n",
|
|
" {\"input\": [\"GraphRag is an amazing LLM framework.\"]},\n",
|
|
"]\n",
|
|
"\n",
|
|
"responses = llm_embedding.embedding_batch(embedding_requests, concurrency=4)\n",
|
|
"for response in responses:\n",
|
|
" if isinstance(response, Exception):\n",
|
|
" print(f\"Error: {response}\")\n",
|
|
" else:\n",
|
|
" for embedding in response.embeddings:\n",
|
|
" print(f\"Embedding vector length: {len(embedding)}\")\n",
|
|
" print(embedding[0:5]) # Print first 5 dimensions of the embedding vector\n",
|
|
"\n",
|
|
"print(f\"Metrics for: {llm_embedding.metrics_store.id}\")\n",
|
|
"print(json.dumps(llm_embedding.metrics_store.get_metrics(), indent=2))"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "0ab62eca",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Details\n",
|
|
"\n",
|
|
"The batch utils start up `concurrency` number of threads in a thread pool and then push all requests into an input queue where free threads pick up the next request to process. The threads will process requests within any defined rate limits and retry any failed request according to the retry settings. If a request fails after all the retries the thread will capture the exception and return it. Thus the batch result may contain exceptions.\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "005ee408",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Thread Pool\n",
|
|
"\n",
|
|
"The batch utils are convenient if all your requests are loaded in memory. If you wish to stream over an input source then you can use the lower level thread pool utils.\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "b4a6553c",
|
|
"metadata": {},
|
|
"source": [
|
|
"### Completion Thread Pool\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 6,
|
|
"id": "05643c93",
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"request_number_1: Succeeded\n",
|
|
"request_number_2: Succeeded\n",
|
|
"request_number_6: Succeeded\n",
|
|
"request_number_9: Succeeded\n",
|
|
"request_number_0: Succeeded\n",
|
|
"request_number_5: Succeeded\n",
|
|
"request_number_7: Succeeded\n",
|
|
"request_number_4: Succeeded\n",
|
|
"request_number_3: Succeeded\n",
|
|
"request_number_8: Succeeded\n",
|
|
"Metrics for: azure/gpt-4o\n",
|
|
"{\n",
|
|
" \"attempted_request_count\": 10,\n",
|
|
" \"successful_response_count\": 10,\n",
|
|
" \"failed_response_count\": 0,\n",
|
|
" \"failure_rate\": 0.0,\n",
|
|
" \"requests_with_retries\": 0,\n",
|
|
" \"retries\": 0,\n",
|
|
" \"retry_rate\": 0.0,\n",
|
|
" \"compute_duration_seconds\": 107.33663082122803,\n",
|
|
" \"compute_duration_per_response_seconds\": 10.733663082122803,\n",
|
|
" \"runtime_duration_seconds\": 0.04277801513671875,\n",
|
|
" \"cached_responses\": 10,\n",
|
|
" \"streaming_responses\": 0,\n",
|
|
" \"responses_with_tokens\": 10,\n",
|
|
" \"prompt_tokens\": 280,\n",
|
|
" \"completion_tokens\": 9240,\n",
|
|
" \"total_tokens\": 9520,\n",
|
|
" \"tokens_per_response\": 952.0,\n",
|
|
" \"responses_with_cost\": 10,\n",
|
|
" \"input_cost\": 0.0007,\n",
|
|
" \"output_cost\": 0.0924,\n",
|
|
" \"total_cost\": 0.0931,\n",
|
|
" \"cost_per_response\": 0.00931\n",
|
|
"}\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"from collections.abc import Iterator\n",
|
|
"\n",
|
|
"from graphrag_llm.types import LLMCompletionChunk, LLMCompletionResponse\n",
|
|
"\n",
|
|
"llm_completion.metrics_store.clear_metrics()\n",
|
|
"\n",
|
|
"\n",
|
|
"# The response handler may also be asynchronous if needed\n",
|
|
"def _handle_response(\n",
|
|
" request_id: str,\n",
|
|
" resp: LLMCompletionResponse | Iterator[LLMCompletionChunk] | Exception,\n",
|
|
"):\n",
|
|
" # Imagine streaming responses to disk or elsewhere\n",
|
|
" if isinstance(resp, Exception):\n",
|
|
" print(f\"{request_id}: Failed\")\n",
|
|
" else:\n",
|
|
" print(f\"{request_id}: Succeeded\")\n",
|
|
"\n",
|
|
"\n",
|
|
"with llm_completion.completion_thread_pool(\n",
|
|
" response_handler=_handle_response,\n",
|
|
" concurrency=25,\n",
|
|
" # set queue_limit to create backpressure on reading the requests\n",
|
|
" queue_limit=10,\n",
|
|
") as completion:\n",
|
|
" # Iterating over a list of completion requests already in memory\n",
|
|
" # but can imagine reading them from disk or another source\n",
|
|
" # The completion function returned from the context manager\n",
|
|
" # will block if the queue_limit is reached until some requests complete\n",
|
|
" # and also requires a request_id for tracking the requests\n",
|
|
" # and allowing you to identify them in the response handler\n",
|
|
" for index, request in enumerate(completion_requests):\n",
|
|
" completion(request_id=f\"request_number_{index}\", **request)\n",
|
|
"\n",
|
|
"# Using the same request that was used in the caching example so\n",
|
|
"# this should complete instantly from cache\n",
|
|
"print(f\"Metrics for: {llm_completion.metrics_store.id}\")\n",
|
|
"print(json.dumps(llm_completion.metrics_store.get_metrics(), indent=2))"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "6e254d56",
|
|
"metadata": {},
|
|
"source": [
|
|
"### Embedding Thread Pool\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 7,
|
|
"id": "7eed1a15",
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"embedding_request_number_1: Succeeded\n",
|
|
"embedding_request_number_0: Succeeded\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"from graphrag_llm.types import LLMEmbeddingResponse\n",
|
|
"\n",
|
|
"llm_embedding.metrics_store.clear_metrics()\n",
|
|
"\n",
|
|
"\n",
|
|
"# The response handler may also be asynchronous if needed\n",
|
|
"def _handle_response(\n",
|
|
" request_id: str,\n",
|
|
" resp: LLMEmbeddingResponse | Exception,\n",
|
|
"):\n",
|
|
" if isinstance(resp, Exception):\n",
|
|
" print(f\"{request_id}: Failed\")\n",
|
|
" else:\n",
|
|
" print(f\"{request_id}: Succeeded\")\n",
|
|
"\n",
|
|
"\n",
|
|
"with llm_embedding.embedding_thread_pool(\n",
|
|
" response_handler=_handle_response,\n",
|
|
" concurrency=25,\n",
|
|
" queue_limit=10,\n",
|
|
") as embedding:\n",
|
|
" for index, request in enumerate(embedding_requests):\n",
|
|
" embedding(request_id=f\"embedding_request_number_{index}\", **request)"
|
|
]
|
|
}
|
|
],
|
|
"metadata": {
|
|
"kernelspec": {
|
|
"display_name": "Python 3",
|
|
"language": "python",
|
|
"name": "python3"
|
|
},
|
|
"language_info": {
|
|
"codemirror_mode": {
|
|
"name": "ipython",
|
|
"version": 3
|
|
},
|
|
"file_extension": ".py",
|
|
"mimetype": "text/x-python",
|
|
"name": "python",
|
|
"nbconvert_exporter": "python",
|
|
"pygments_lexer": "ipython3",
|
|
"version": "3.11.9"
|
|
}
|
|
},
|
|
"nbformat": 4,
|
|
"nbformat_minor": 5
|
|
}
|