chore: import upstream snapshot with attribution

docs-website/reference_versioned_docs/version-2.18/integrations-api/watsonx.md

@@ -0,0 +1,701 @@
+---
+title: "IBM watsonx.ai"
+id: integrations-watsonx
+description: "IBM watsonx.ai integration for Haystack"
+slug: "/integrations-watsonx"
+---
+
+
+## haystack_integrations.components.embedders.watsonx.document_embedder
+
+### WatsonxDocumentEmbedder
+
+Computes document embeddings using IBM watsonx.ai models.
+
+### Usage example
+
+```python
+from haystack import Document
+from haystack_integrations.components.embedders.watsonx.document_embedder import WatsonxDocumentEmbedder
+
+documents = [
+    Document(content="I love pizza!"),
+    Document(content="Pasta is great too"),
+]
+
+document_embedder = WatsonxDocumentEmbedder(
+    model="ibm/slate-30m-english-rtrvr-v2",
+    api_key=Secret.from_env_var("WATSONX_API_KEY"),
+    api_base_url="https://us-south.ml.cloud.ibm.com",
+    project_id=Secret.from_env_var("WATSONX_PROJECT_ID"),
+)
+
+result = document_embedder.run(documents=documents)
+print(result["documents"][0].embedding)
+
+# [0.017020374536514282, -0.023255806416273117, ...]
+```
+
+#### __init__
+
+```python
+__init__(
+    *,
+    model: str = "ibm/slate-30m-english-rtrvr-v2",
+    api_key: Secret = Secret.from_env_var("WATSONX_API_KEY"),
+    api_base_url: str = "https://us-south.ml.cloud.ibm.com",
+    project_id: Secret = Secret.from_env_var("WATSONX_PROJECT_ID"),
+    truncate_input_tokens: int | None = None,
+    prefix: str = "",
+    suffix: str = "",
+    batch_size: int = 1000,
+    concurrency_limit: int = 5,
+    timeout: float | None = None,
+    max_retries: int | None = None,
+    meta_fields_to_embed: list[str] | None = None,
+    embedding_separator: str = "\n"
+) -> None
+```
+
+Creates a WatsonxDocumentEmbedder component.
+
+**Parameters:**
+
+- **model** (<code>str</code>) – The name of the model to use for calculating embeddings.
+  Default is "ibm/slate-30m-english-rtrvr-v2".
+- **api_key** (<code>Secret</code>) – The WATSONX API key. Can be set via environment variable WATSONX_API_KEY.
+- **api_base_url** (<code>str</code>) – The WATSONX URL for the watsonx.ai service.
+  Default is "https://us-south.ml.cloud.ibm.com".
+- **project_id** (<code>Secret</code>) – The ID of the Watson Studio project.
+  Can be set via environment variable WATSONX_PROJECT_ID.
+- **truncate_input_tokens** (<code>int | None</code>) – Maximum number of tokens to use from the input text.
+  If set to `None` (or not provided), the full input text is used, up to the model's maximum token limit.
+- **prefix** (<code>str</code>) – A string to add at the beginning of each text.
+- **suffix** (<code>str</code>) – A string to add at the end of each text.
+- **batch_size** (<code>int</code>) – Number of documents to embed in one API call. Default is 1000.
+- **concurrency_limit** (<code>int</code>) – Number of parallel requests to make. Default is 5.
+- **timeout** (<code>float | None</code>) – Timeout for API requests in seconds.
+- **max_retries** (<code>int | None</code>) – Maximum number of retries for API requests.
+
+#### to_dict
+
+```python
+to_dict() -> dict[str, Any]
+```
+
+Serialize the component to a dictionary.
+
+**Returns:**
+
+- <code>dict\[str, Any\]</code> – The serialized component as a dictionary.
+
+#### from_dict
+
+```python
+from_dict(data: dict[str, Any]) -> 'WatsonxDocumentEmbedder'
+```
+
+Deserializes the component from a dictionary.
+
+**Parameters:**
+
+- **data** (<code>dict\[str, Any\]</code>) – The dictionary representation of this component.
+
+**Returns:**
+
+- <code>'WatsonxDocumentEmbedder'</code> – The deserialized component instance.
+
+#### run
+
+```python
+run(documents: list[Document]) -> dict[str, list[Document] | dict[str, Any]]
+```
+
+Embeds a list of documents.
+
+**Parameters:**
+
+- **documents** (<code>list\[Document\]</code>) – A list of documents to embed.
+
+**Returns:**
+
+- <code>dict\[str, list\[Document\] | dict\[str, Any\]\]</code> – A dictionary with:
+- 'documents': List of Documents with embeddings added
+- 'meta': Information about the model usage
+
+## haystack_integrations.components.embedders.watsonx.text_embedder
+
+### WatsonxTextEmbedder
+
+Embeds strings using IBM watsonx.ai foundation models.
+
+You can use it to embed user query and send it to an embedding Retriever.
+
+### Usage example
+
+```python
+from haystack_integrations.components.embedders.watsonx.text_embedder import WatsonxTextEmbedder
+
+text_to_embed = "I love pizza!"
+
+text_embedder = WatsonxTextEmbedder(
+    model="ibm/slate-30m-english-rtrvr-v2",
+    api_key=Secret.from_env_var("WATSONX_API_KEY"),
+    api_base_url="https://us-south.ml.cloud.ibm.com",
+    project_id=Secret.from_env_var("WATSONX_PROJECT_ID"),
+)
+
+print(text_embedder.run(text_to_embed))
+
+# {'embedding': [0.017020374536514282, -0.023255806416273117, ...],
+#  'meta': {'model': 'ibm/slate-30m-english-rtrvr-v2',
+#           'truncated_input_tokens': 3}}
+```
+
+#### __init__
+
+```python
+__init__(
+    *,
+    model: str = "ibm/slate-30m-english-rtrvr-v2",
+    api_key: Secret = Secret.from_env_var("WATSONX_API_KEY"),
+    api_base_url: str = "https://us-south.ml.cloud.ibm.com",
+    project_id: Secret = Secret.from_env_var("WATSONX_PROJECT_ID"),
+    truncate_input_tokens: int | None = None,
+    prefix: str = "",
+    suffix: str = "",
+    timeout: float | None = None,
+    max_retries: int | None = None
+) -> None
+```
+
+Creates an WatsonxTextEmbedder component.
+
+**Parameters:**
+
+- **model** (<code>str</code>) – The name of the IBM watsonx model to use for calculating embeddings.
+  Default is "ibm/slate-30m-english-rtrvr-v2".
+- **api_key** (<code>Secret</code>) – The WATSONX API key. Can be set via environment variable WATSONX_API_KEY.
+- **api_base_url** (<code>str</code>) – The WATSONX URL for the watsonx.ai service.
+  Default is "https://us-south.ml.cloud.ibm.com".
+- **project_id** (<code>Secret</code>) – The ID of the Watson Studio project.
+  Can be set via environment variable WATSONX_PROJECT_ID.
+- **truncate_input_tokens** (<code>int | None</code>) – Maximum number of tokens to use from the input text.
+  If set to `None` (or not provided), the full input text is used, up to the model's maximum token limit.
+- **prefix** (<code>str</code>) – A string to add at the beginning of each text to embed.
+- **suffix** (<code>str</code>) – A string to add at the end of each text to embed.
+- **timeout** (<code>float | None</code>) – Timeout for API requests in seconds.
+- **max_retries** (<code>int | None</code>) – Maximum number of retries for API requests.
+
+#### to_dict
+
+```python
+to_dict() -> dict[str, Any]
+```
+
+Serialize the component to a dictionary.
+
+**Returns:**
+
+- <code>dict\[str, Any\]</code> – The serialized component as a dictionary.
+
+#### from_dict
+
+```python
+from_dict(data: dict[str, Any]) -> WatsonxTextEmbedder
+```
+
+Deserializes the component from a dictionary.
+
+**Parameters:**
+
+- **data** (<code>dict\[str, Any\]</code>) – The dictionary representation of this component.
+
+**Returns:**
+
+- <code>WatsonxTextEmbedder</code> – The deserialized component instance.
+
+#### run
+
+```python
+run(text: str) -> dict[str, list[float] | dict[str, Any]]
+```
+
+Embeds a single string.
+
+**Parameters:**
+
+- **text** (<code>str</code>) – Text to embed.
+
+**Returns:**
+
+- <code>dict\[str, list\[float\] | dict\[str, Any\]\]</code> – A dictionary with:
+- 'embedding': The embedding of the input text
+- 'meta': Information about the model usage
+
+## haystack_integrations.components.generators.watsonx.chat.chat_generator
+
+### WatsonxChatGenerator
+
+Enables chat completions using IBM's watsonx.ai foundation models.
+
+This component interacts with IBM's watsonx.ai platform to generate chat responses using various foundation
+models. It supports the [ChatMessage](https://docs.haystack.deepset.ai/docs/chatmessage) format for both input
+and output, including multimodal inputs with text and images.
+
+The generator works with IBM's foundation models that are listed
+[here](https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/fm-models.html?context=wx&audience=wdp).
+
+You can customize the generation behavior by passing parameters to the watsonx.ai API through the
+`generation_kwargs` argument. These parameters are passed directly to the watsonx.ai inference endpoint.
+
+For details on watsonx.ai API parameters, see
+[IBM watsonx.ai documentation](https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/fm-parameters.html).
+
+### Usage example
+
+```python
+from haystack_integrations.components.generators.watsonx.chat.chat_generator import WatsonxChatGenerator
+from haystack.dataclasses import ChatMessage
+from haystack.utils import Secret
+
+messages = [ChatMessage.from_user("Explain quantum computing in simple terms")]
+
+client = WatsonxChatGenerator(
+    api_key=Secret.from_env_var("WATSONX_API_KEY"),
+    model="ibm/granite-4-h-small",
+    project_id=Secret.from_env_var("WATSONX_PROJECT_ID"),
+)
+response = client.run(messages)
+print(response)
+```
+
+### Multimodal usage example
+
+```python
+from haystack.dataclasses import ChatMessage, ImageContent
+
+# Create an image from file path or base64
+image_content = ImageContent.from_file_path("path/to/your/image.jpg")
+
+# Create a multimodal message with both text and image
+messages = [ChatMessage.from_user(content_parts=["What's in this image?", image_content])]
+
+# Use a multimodal model
+client = WatsonxChatGenerator(
+    api_key=Secret.from_env_var("WATSONX_API_KEY"),
+    model="meta-llama/llama-3-2-11b-vision-instruct",
+    project_id=Secret.from_env_var("WATSONX_PROJECT_ID"),
+)
+response = client.run(messages)
+print(response)
+```
+
+#### SUPPORTED_MODELS
+
+```python
+SUPPORTED_MODELS: list[str] = [
+    "ibm/granite-3-1-8b-base",
+    "ibm/granite-3-8b-instruct",
+    "ibm/granite-4-h-small",
+    "ibm/granite-8b-code-instruct",
+    "ibm/granite-guardian-3-8b",
+    "meta-llama/llama-3-1-70b-gptq",
+    "meta-llama/llama-3-1-8b",
+    "meta-llama/llama-3-2-11b-vision-instruct",
+    "meta-llama/llama-3-2-90b-vision-instruct",
+    "meta-llama/llama-3-3-70b-instruct",
+    "meta-llama/llama-3-405b-instruct",
+    "meta-llama/llama-4-maverick-17b-128e-instruct-fp8",
+    "meta-llama/llama-guard-3-11b-vision",
+    "mistral-large-2512",
+    "mistralai/mistral-medium-2505",
+    "mistralai/mistral-small-3-1-24b-instruct-2503",
+    "openai/gpt-oss-120b",
+]
+
+```
+
+A non-exhaustive list of models supported by this component.
+
+See https://www.ibm.com/docs/en/watsonx/saas?topic=solutions-supported-foundation-models for the
+full list of models and up-to-date model IDs.
+
+#### __init__
+
+```python
+__init__(
+    *,
+    api_key: Secret = Secret.from_env_var("WATSONX_API_KEY"),
+    model: str = "ibm/granite-4-h-small",
+    project_id: Secret = Secret.from_env_var("WATSONX_PROJECT_ID"),
+    api_base_url: str = "https://us-south.ml.cloud.ibm.com",
+    generation_kwargs: dict[str, Any] | None = None,
+    timeout: float | None = None,
+    max_retries: int | None = None,
+    verify: bool | str | None = None,
+    streaming_callback: StreamingCallbackT | None = None,
+    tools: ToolsType | None = None
+) -> None
+```
+
+Creates an instance of WatsonxChatGenerator.
+
+Before initializing the component, you can set environment variables:
+
+- `WATSONX_TIMEOUT` to override the default timeout
+- `WATSONX_MAX_RETRIES` to override the default retry count
+
+**Parameters:**
+
+- **api_key** (<code>Secret</code>) – IBM Cloud API key for watsonx.ai access.
+  Can be set via `WATSONX_API_KEY` environment variable or passed directly.
+- **model** (<code>str</code>) – The model ID to use for completions. Defaults to "ibm/granite-4-h-small".
+  Available models can be found in your IBM Cloud account.
+- **project_id** (<code>Secret</code>) – IBM Cloud project ID
+- **api_base_url** (<code>str</code>) – Custom base URL for the API endpoint.
+  Defaults to "https://us-south.ml.cloud.ibm.com".
+- **generation_kwargs** (<code>dict\[str, Any\] | None</code>) – Additional parameters to control text generation.
+  These parameters are passed directly to the watsonx.ai inference endpoint.
+  Supported parameters include:
+- `temperature`: Controls randomness (lower = more deterministic)
+- `max_new_tokens`: Maximum number of tokens to generate
+- `min_new_tokens`: Minimum number of tokens to generate
+- `top_p`: Nucleus sampling probability threshold
+- `top_k`: Number of highest probability tokens to consider
+- `repetition_penalty`: Penalty for repeated tokens
+- `length_penalty`: Penalty based on output length
+- `stop_sequences`: List of sequences where generation should stop
+- `random_seed`: Seed for reproducible results
+- **timeout** (<code>float | None</code>) – Timeout in seconds for API requests.
+  Defaults to environment variable `WATSONX_TIMEOUT` or 30 seconds.
+- **max_retries** (<code>int | None</code>) – Maximum number of retry attempts for failed requests.
+  Defaults to environment variable `WATSONX_MAX_RETRIES` or 5.
+- **verify** (<code>bool | str | None</code>) – SSL verification setting. Can be:
+- True: Verify SSL certificates (default)
+- False: Skip verification (insecure)
+- Path to CA bundle for custom certificates
+- **streaming_callback** (<code>StreamingCallbackT | None</code>) – A callback function for streaming responses.
+- **tools** (<code>ToolsType | None</code>) – A list of Tool and/or Toolset objects, or a single Toolset for which the model can prepare calls.
+
+#### to_dict
+
+```python
+to_dict() -> dict[str, Any]
+```
+
+Serialize the component to a dictionary.
+
+**Returns:**
+
+- <code>dict\[str, Any\]</code> – The serialized component as a dictionary.
+
+#### from_dict
+
+```python
+from_dict(data: dict[str, Any]) -> WatsonxChatGenerator
+```
+
+Deserialize this component from a dictionary.
+
+**Parameters:**
+
+- **data** (<code>dict\[str, Any\]</code>) – The dictionary representation of this component.
+
+**Returns:**
+
+- <code>WatsonxChatGenerator</code> – The deserialized component instance.
+
+#### run
+
+```python
+run(
+    *,
+    messages: list[ChatMessage] | str,
+    generation_kwargs: dict[str, Any] | None = None,
+    streaming_callback: StreamingCallbackT | None = None,
+    tools: ToolsType | None = None
+) -> dict[str, list[ChatMessage]]
+```
+
+Generate chat completions synchronously.
+
+**Parameters:**
+
+- **messages** (<code>list\[ChatMessage\] | str</code>) – A list of ChatMessage instances representing the input messages.
+  If a string is provided, it is converted to a list containing a ChatMessage with user role.
+- **generation_kwargs** (<code>dict\[str, Any\] | None</code>) – Additional keyword arguments for text generation. These parameters will potentially override the parameters
+  passed in the `__init__` method.
+- **streaming_callback** (<code>StreamingCallbackT | None</code>) – A callback function that is called when a new token is received from the stream.
+  If provided this will override the `streaming_callback` set in the `__init__` method.
+- **tools** (<code>ToolsType | None</code>) – A list of Tool and/or Toolset objects, or a single Toolset for which the model can prepare calls.
+  If set, it will override the `tools` parameter provided during initialization.
+
+**Returns:**
+
+- <code>dict\[str, list\[ChatMessage\]\]</code> – A dictionary with the following key:
+- `replies`: A list containing the generated responses as ChatMessage instances.
+
+#### run_async
+
+```python
+run_async(
+    *,
+    messages: list[ChatMessage] | str,
+    generation_kwargs: dict[str, Any] | None = None,
+    streaming_callback: StreamingCallbackT | None = None,
+    tools: ToolsType | None = None
+) -> dict[str, list[ChatMessage]]
+```
+
+Generate chat completions asynchronously.
+
+**Parameters:**
+
+- **messages** (<code>list\[ChatMessage\] | str</code>) – A list of ChatMessage instances representing the input messages.
+  If a string is provided, it is converted to a list containing a ChatMessage with user role.
+- **generation_kwargs** (<code>dict\[str, Any\] | None</code>) – Additional keyword arguments for text generation. These parameters will potentially override the parameters
+  passed in the `__init__` method.
+- **streaming_callback** (<code>StreamingCallbackT | None</code>) – A callback function that is called when a new token is received from the stream.
+  If provided this will override the `streaming_callback` set in the `__init__` method.
+- **tools** (<code>ToolsType | None</code>) – A list of Tool and/or Toolset objects, or a single Toolset for which the model can prepare calls.
+  If set, it will override the `tools` parameter provided during initialization.
+
+**Returns:**
+
+- <code>dict\[str, list\[ChatMessage\]\]</code> – A dictionary with the following key:
+- `replies`: A list containing the generated responses as ChatMessage instances.
+
+## haystack_integrations.components.generators.watsonx.generator
+
+### WatsonxGenerator
+
+Bases: <code>WatsonxChatGenerator</code>
+
+Enables text completions using IBM's watsonx.ai foundation models.
+
+This component extends WatsonxChatGenerator to provide the standard Generator interface that works with prompt
+strings instead of ChatMessage objects.
+
+The generator works with IBM's foundation models that are listed
+[here](https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/fm-models.html?context=wx&audience=wdp).
+
+You can customize the generation behavior by passing parameters to the watsonx.ai API through the
+`generation_kwargs` argument. These parameters are passed directly to the watsonx.ai inference endpoint.
+
+For details on watsonx.ai API parameters, see
+[IBM watsonx.ai documentation](https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/fm-parameters.html).
+
+### Usage example
+
+```python
+from haystack_integrations.components.generators.watsonx.generator import WatsonxGenerator
+from haystack.utils import Secret
+
+generator = WatsonxGenerator(
+    api_key=Secret.from_env_var("WATSONX_API_KEY"),
+    model="ibm/granite-4-h-small",
+    project_id=Secret.from_env_var("WATSONX_PROJECT_ID"),
+)
+
+response = generator.run(
+    prompt="Explain quantum computing in simple terms",
+    system_prompt="You are a helpful physics teacher.",
+)
+print(response)
+```
+
+Output:
+
+```
+{
+    "replies": ["Quantum computing uses quantum-mechanical phenomena like...."],
+    "meta": [
+        {
+            "model": "ibm/granite-4-h-small",
+            "project_id": "your-project-id",
+            "usage": {
+                "prompt_tokens": 12,
+                "completion_tokens": 45,
+                "total_tokens": 57,
+            },
+        }
+    ],
+}
+```
+
+#### SUPPORTED_MODELS
+
+```python
+SUPPORTED_MODELS: list[str] = [
+    "ibm/granite-3-1-8b-base",
+    "ibm/granite-3-8b-instruct",
+    "ibm/granite-4-h-small",
+    "ibm/granite-8b-code-instruct",
+    "ibm/granite-guardian-3-8b",
+    "meta-llama/llama-3-1-70b-gptq",
+    "meta-llama/llama-3-1-8b",
+    "meta-llama/llama-3-2-11b-vision-instruct",
+    "meta-llama/llama-3-2-90b-vision-instruct",
+    "meta-llama/llama-3-3-70b-instruct",
+    "meta-llama/llama-3-405b-instruct",
+    "meta-llama/llama-4-maverick-17b-128e-instruct-fp8",
+    "meta-llama/llama-guard-3-11b-vision",
+    "mistral-large-2512",
+    "mistralai/mistral-medium-2505",
+    "mistralai/mistral-small-3-1-24b-instruct-2503",
+    "openai/gpt-oss-120b",
+]
+
+```
+
+A non-exhaustive list of models supported by this component.
+
+See https://www.ibm.com/docs/en/watsonx/saas?topic=solutions-supported-foundation-models for the
+full list of models and up-to-date model IDs.
+
+#### __init__
+
+```python
+__init__(
+    *,
+    api_key: Secret = Secret.from_env_var("WATSONX_API_KEY"),
+    model: str = "ibm/granite-4-h-small",
+    project_id: Secret = Secret.from_env_var("WATSONX_PROJECT_ID"),
+    api_base_url: str = "https://us-south.ml.cloud.ibm.com",
+    system_prompt: str | None = None,
+    generation_kwargs: dict[str, Any] | None = None,
+    timeout: float | None = None,
+    max_retries: int | None = None,
+    verify: bool | str | None = None,
+    streaming_callback: StreamingCallbackT | None = None
+) -> None
+```
+
+Creates an instance of WatsonxGenerator.
+
+Before initializing the component, you can set environment variables:
+
+- `WATSONX_TIMEOUT` to override the default timeout
+- `WATSONX_MAX_RETRIES` to override the default retry count
+
+**Parameters:**
+
+- **api_key** (<code>Secret</code>) – IBM Cloud API key for watsonx.ai access.
+  Can be set via `WATSONX_API_KEY` environment variable or passed directly.
+- **model** (<code>str</code>) – The model ID to use for completions. Defaults to "ibm/granite-4-h-small".
+  Available models can be found in your IBM Cloud account.
+- **project_id** (<code>Secret</code>) – IBM Cloud project ID
+- **api_base_url** (<code>str</code>) – Custom base URL for the API endpoint.
+  Defaults to "https://us-south.ml.cloud.ibm.com".
+- **system_prompt** (<code>str | None</code>) – The system prompt to use for text generation.
+- **generation_kwargs** (<code>dict\[str, Any\] | None</code>) – Additional parameters to control text generation.
+  These parameters are passed directly to the watsonx.ai inference endpoint.
+  Supported parameters include:
+- `temperature`: Controls randomness (lower = more deterministic)
+- `max_new_tokens`: Maximum number of tokens to generate
+- `min_new_tokens`: Minimum number of tokens to generate
+- `top_p`: Nucleus sampling probability threshold
+- `top_k`: Number of highest probability tokens to consider
+- `repetition_penalty`: Penalty for repeated tokens
+- `length_penalty`: Penalty based on output length
+- `stop_sequences`: List of sequences where generation should stop
+- `random_seed`: Seed for reproducible results
+- **timeout** (<code>float | None</code>) – Timeout in seconds for API requests.
+  Defaults to environment variable `WATSONX_TIMEOUT` or 30 seconds.
+- **max_retries** (<code>int | None</code>) – Maximum number of retry attempts for failed requests.
+  Defaults to environment variable `WATSONX_MAX_RETRIES` or 5.
+- **verify** (<code>bool | str | None</code>) – SSL verification setting. Can be:
+- True: Verify SSL certificates (default)
+- False: Skip verification (insecure)
+- Path to CA bundle for custom certificates
+- **streaming_callback** (<code>StreamingCallbackT | None</code>) – A callback function for streaming responses.
+
+#### to_dict
+
+```python
+to_dict() -> dict[str, Any]
+```
+
+Serialize the component to a dictionary.
+
+**Returns:**
+
+- <code>dict\[str, Any\]</code> – The serialized component as a dictionary.
+
+#### from_dict
+
+```python
+from_dict(data: dict[str, Any]) -> WatsonxGenerator
+```
+
+Deserialize this component from a dictionary.
+
+**Parameters:**
+
+- **data** (<code>dict\[str, Any\]</code>) – The dictionary representation of this component.
+
+**Returns:**
+
+- <code>WatsonxGenerator</code> – The deserialized component instance.
+
+#### run
+
+```python
+run(
+    *,
+    prompt: str,
+    system_prompt: str | None = None,
+    streaming_callback: StreamingCallbackT | None = None,
+    generation_kwargs: dict[str, Any] | None = None
+) -> dict[str, Any]
+```
+
+Generate text completions synchronously.
+
+**Parameters:**
+
+- **prompt** (<code>str</code>) – The input prompt string for text generation.
+- **system_prompt** (<code>str | None</code>) – An optional system prompt to provide context or instructions for the generation.
+  If not provided, the system prompt set in the `__init__` method will be used.
+- **streaming_callback** (<code>StreamingCallbackT | None</code>) – A callback function that is called when a new token is received from the stream.
+  If provided, this will override the `streaming_callback` set in the `__init__` method.
+- **generation_kwargs** (<code>dict\[str, Any\] | None</code>) – Additional keyword arguments for text generation. These parameters will potentially override the parameters
+  passed in the `__init__` method. Supported parameters include temperature, max_new_tokens, top_p, etc.
+
+**Returns:**
+
+- <code>dict\[str, Any\]</code> – A dictionary with the following keys:
+- `replies`: A list of generated text completions as strings.
+- `meta`: A list of metadata dictionaries containing information about each generation,
+  including model name, finish reason, and token usage statistics.
+
+#### run_async
+
+```python
+run_async(
+    *,
+    prompt: str,
+    system_prompt: str | None = None,
+    streaming_callback: StreamingCallbackT | None = None,
+    generation_kwargs: dict[str, Any] | None = None
+) -> dict[str, Any]
+```
+
+Generate text completions asynchronously.
+
+**Parameters:**
+
+- **prompt** (<code>str</code>) – The input prompt string for text generation.
+- **system_prompt** (<code>str | None</code>) – An optional system prompt to provide context or instructions for the generation.
+- **streaming_callback** (<code>StreamingCallbackT | None</code>) – A callback function that is called when a new token is received from the stream.
+  If provided, this will override the `streaming_callback` set in the `__init__` method.
+- **generation_kwargs** (<code>dict\[str, Any\] | None</code>) – Additional keyword arguments for text generation. These parameters will potentially override the parameters
+  passed in the `__init__` method. Supported parameters include temperature, max_new_tokens, top_p, etc.
+
+**Returns:**
+
+- <code>dict\[str, Any\]</code> – A dictionary with the following keys:
+- `replies`: A list of generated text completions as strings.
+- `meta`: A list of metadata dictionaries containing information about each generation,
+  including model name, finish reason, and token usage statistics.