chore: import upstream snapshot with attribution

docs-website/docs/development/deployment.mdx

@@ -0,0 +1,36 @@
+---
+title: "Deployment"
+id: deployment
+slug: "/deployment"
+description: "Deploy your Haystack pipelines through various services such as Docker, Kubernetes, Ray, or a variety of Serverless options."
+---
+
+# Deployment
+
+Deploy your Haystack pipelines through various services such as Docker, Kubernetes, Ray, or a variety of Serverless options.
+
+As a framework, Haystack is typically integrated into a variety of applications and environments, and there is no single, specific deployment strategy to follow. However, it is very common to make Haystack pipelines accessible through a service that can be easily called from other software systems.
+
+These guides focus on tools and techniques that can be used to run Haystack pipelines in common scenarios. While these suggestions should not be considered the only way to do so, they should provide inspiration and the ability to customize them according to your needs.
+
+### Guides
+
+Here are the currently available guides on Haystack pipeline deployment:
+
+- [Deploying with Docker](deployment/docker.mdx)
+- [Deploying with Kubernetes](deployment/kubernetes.mdx)
+- [Deploying with OpenShift](deployment/openshift.mdx)
+
+### Hayhooks
+
+Haystack can be easily integrated into any HTTP application, but if you don’t have one, you can use Hayhooks, a ready-made application that serves Haystack pipelines as REST endpoints. We’ll be using Hayhooks throughout this guide to streamline the code examples. Refer to the Hayhooks [overview](hayhooks.mdx) for a quick start, or the [official Hayhooks documentation](https://deepset-ai.github.io/hayhooks/) for comprehensive guides and reference.
+
+:::note[Looking to scale with confidence?]
+
+If your team needs **enterprise-grade support, best practices, and deployment guidance** to run Haystack in production, check out **Haystack Enterprise Starter**.
+
+📜 [Learn more about Haystack Enterprise Starter](https://haystack.deepset.ai/blog/announcing-haystack-enterprise)  
+🤝 [Get in touch with our team](https://www.deepset.ai/products-and-services/haystack-enterprise-starter)
+
+👉 For platform tooling to **manage data, pipelines, testing, and governance at scale**, explore the [Haystack Enterprise Platform](https://www.deepset.ai/products-and-services/haystack-enterprise-platform).
+:::

docs-website/docs/development/deployment/docker.mdx

@@ -0,0 +1,117 @@
+---
+title: "Docker"
+id: docker
+slug: "/docker"
+description: "Learn how to deploy your Haystack pipelines through Docker starting from the basic Docker container to a complex application using Hayhooks."
+---
+
+# Docker
+
+Learn how to deploy your Haystack pipelines through Docker starting from the basic Docker container to a complex application using Hayhooks.
+
+## Running Haystack in Docker
+
+The most basic form of Haystack deployment happens through Docker containers. Becoming familiar with running and customizing Haystack Docker images is useful as they form the basis for more advanced deployment.
+
+Haystack releases are officially distributed through the [`deepset/haystack`](https://hub.docker.com/r/deepset/haystack) Docker image. Haystack images come in different flavors depending on the specific components they ship and the Haystack version.
+
+:::info
+At the moment, the only flavor available for Haystack is `base`, which ships exactly what you would get by installing Haystack locally with `pip install haystack-ai`.
+:::
+
+You can pull a specific Haystack flavor using Docker tags: for example, to pull the image containing Haystack `2.12.1`, you can run the command:
+
+```shell
+docker pull deepset/haystack:base-v2.12.1
+```
+
+Although the `base` flavor is meant to be customized, it can also be used to quickly run Haystack scripts locally without the need to set up a Python environment and its dependencies. For example, this is how you would print Haystack’s version running a Docker container:
+
+```shell
+docker run -it --rm deepset/haystack:base-v2.12.1 python -c"from haystack.version import __version__; print(__version__)"
+```
+
+## Customizing the Haystack Docker Image
+
+Chances are your application will be more complex than a simple script, and you’ll need to install additional dependencies inside the Docker image along with Haystack.
+
+For example, you might want to run a simple indexing pipeline using [Chroma](../../document-stores/chromadocumentstore.mdx) as your Document Store using a Docker container. The `base` image only contains a basic install of Haystack, but you need to install the Chroma integration (`chroma-haystack`) package additionally. The best approach would be to create a custom Docker image shipping the extra dependency.
+
+Assuming you have a `main.py` script in your current folder, the Dockerfile would look like this:
+
+```shell
+FROM deepset/haystack:base-v2.12.1
+
+RUN pip install chroma-haystack
+
+COPY ./main.py /usr/src/myapp/main.py
+
+ENTRYPOINT ["python", "/usr/src/myapp/main.py"]
+```
+
+Then you can create your custom Haystack image with:
+
+```shell
+docker build . -t my-haystack-image
+```
+
+## Complex Application with Docker Compose
+
+A Haystack application running in Docker can go pretty far: with an internet connection, the container can reach external services providing vector databases, inference endpoints, and observability features.
+
+Still, you might want to orchestrate additional services for your Haystack container locally, for example, to reduce costs or increase performance. When your application runtime depends on more than one Docker container, [Docker Compose](https://docs.docker.com/compose/) is a great tool to keep everything together.
+
+As an example, let’s say your application wraps two pipelines: one to _index_ documents into a Qdrant instance and the other to _query_ those documents at a later time. This setup would require two Docker containers: one to run the pipelines as REST APIs using [Hayhooks](../hayhooks.mdx) and a second to run a Qdrant instance. For more information on configuring Hayhooks using Docker Compose, see the [official Hayhooks documentation](https://deepset-ai.github.io/hayhooks/getting-started/quick-start-docker/).
+
+For building the Hayhooks image, we can easily customize the base image of one of the latest versions of Hayhooks, adding required dependencies required by [`QdrantDocumentStore`](../../document-stores/qdrant-document-store.mdx). The Dockerfile would look like this:
+
+```dockerfile Dockerfile
+FROM deepset/hayhooks:v1.16.0
+
+RUN pip install qdrant-haystack sentence-transformers
+
+CMD ["hayhooks", "run", "--host", "0.0.0.0"]
+
+```
+
+We wouldn’t need to customize Qdrant, so their official Docker image would work perfectly. The `docker-compose.yml` file would then look like this:
+
+```yaml
+services:
+  qdrant:
+    image: qdrant/qdrant:latest
+    restart: always
+    container_name: qdrant
+    ports:
+      - 6333:6333
+      - 6334:6334
+    expose:
+      - 6333
+      - 6334
+      - 6335
+    configs:
+      - source: qdrant_config
+        target: /qdrant/config/production.yaml
+    volumes:
+      - ./qdrant_data:/qdrant_data
+
+  hayhooks:
+    build: . # Build from local Dockerfile
+    container_name: hayhooks
+    ports:
+      - "1416:1416"
+    volumes:
+      - ./pipelines:/pipelines
+    environment:
+      - HAYHOOKS_PIPELINES_DIR=/pipelines
+      - LOG=DEBUG
+    depends_on:
+      - qdrant
+
+configs:
+  qdrant_config:
+    content: |
+      log_level: INFO
+```
+
+For a functional example of a Docker Compose deployment, check out the [“RAG indexing and querying with Elasticsearch”](https://github.com/deepset-ai/hayhooks/tree/main/examples/rag_indexing_query) example from GitHub.

docs-website/docs/development/deployment/kubernetes.mdx

@@ -0,0 +1,269 @@
+---
+title: "Kubernetes"
+id: kubernetes
+slug: "/kubernetes"
+description: "Learn how to deploy your Haystack pipelines through Kubernetes."
+---
+
+import ClickableImage from "@site/src/components/ClickableImage";
+
+# Kubernetes
+
+Learn how to deploy your Haystack pipelines through Kubernetes.
+
+The best way to get Haystack running as a workload in a container orchestrator like Kubernetes is to create a service to expose one or more [Hayhooks](../hayhooks.mdx) instances.
+
+## Create a Haystack Kubernetes Service using Hayhooks
+
+As a first step, we recommend to create a local [KinD](https://github.com/kubernetes-sigs/kind) or [Minikube](https://github.com/kubernetes/minikube) Kubernetes cluster. You can manage your cluster from CLI, but tools like [k9s](https://k9scli.io/) or [Lens](https://k8slens.dev/) can ease the process.
+
+When done, start with a very simple Kubernetes Service running a single Hayhooks Pod:
+
+```yaml
+kind: Pod
+apiVersion: v1
+metadata:
+  name: hayhooks
+  labels:
+    app: haystack
+spec:
+  containers:
+    - image: deepset/hayhooks:v1.16.0
+      name: hayhooks
+      imagePullPolicy: IfNotPresent
+      resources:
+        limits:
+          memory: "512Mi"
+          cpu: "500m"
+        requests:
+          memory: "256Mi"
+          cpu: "250m"
+
+---
+
+kind: Service
+apiVersion: v1
+metadata:
+  name: haystack-service
+spec:
+  selector:
+    app: haystack
+  type: ClusterIP
+  ports:
+    # Default port used by the Hayhooks Docker image
+    - port: 1416
+
+```
+
+After applying the above to an existing Kubernetes cluster, a `hayhooks` Pod will show up as a Service called `haystack-service`.
+<ClickableImage src="/img/6eb9fb0c7b00367bfbe8182ffc7c3746f3f3d03b720e963df045e28160362d7f-Screenshot_2025-04-15_at_16.15.28.png" alt="Kubernetes Lens interface showing the hayhooks Pod running in the default namespace with status Running" />
+
+Note that the `Service` defined above is of type `ClusterIP`. That means it's exposed only _inside_ the Kubernetes cluster. To expose the Hayhooks API to the _outside_ world as well, you need a `NodePort` or `Ingress` resource. As an alternative, it's also possible to use [Port Forwarding](https://kubernetes.io/docs/tasks/access-application-cluster/port-forward-access-application-cluster/) to access the `Service` locally.
+
+To do that, add port `30080` to Host-To-Node Mapping of our KinD cluster. In other words, make sure that the cluster is created with a node configuration similar to the following:
+
+```yaml
+kind: Cluster
+apiVersion: kind.x-k8s.io/v1alpha4
+nodes:
+  - role: control-plane
+    # ...
+    extraPortMappings:
+      - containerPort: 30080
+        hostPort: 30080
+        protocol: TCP
+```
+
+Then, create a simple `NodePort`  to test if Hayhooks Pod is running correctly:
+
+```yaml
+apiVersion: v1
+kind: Service
+metadata:
+  name: haystack-nodeport
+spec:
+  selector:
+    app: haystack
+  type: NodePort
+  ports:
+  - port: 1416
+    targetPort: 1416
+    nodePort: 30080
+    name: http
+```
+
+After applying this, `hayhooks` Pod will be accessible on `localhost:30080`.
+
+From here, you should be able to manage pipelines. Remember that it's possible to deploy multiple different pipelines on a single Hayhooks instance. Check the [Hayhooks overview](../hayhooks.mdx) or the [official Hayhooks documentation](https://deepset-ai.github.io/hayhooks/) for more details.
+
+## Auto-Run Pipelines at Pod Start
+
+Hayhooks can load Haystack pipelines at startup, making them readily available when the server starts. You can leverage this mechanism to have your pods immediately serve one or more pipelines when they start.
+
+At startup, it will look for deployed pipelines on the path specified at `HAYHOOKS_PIPELINES_DIR`, then load them.
+
+A [deployed pipeline](https://github.com/deepset-ai/hayhooks?tab=readme-ov-file#deploy-a-pipeline) is essentially a directory which must contain a `pipeline_wrapper.py` file and possibly other files. To preload an [example pipeline](https://github.com/deepset-ai/hayhooks/tree/main/examples/pipeline_wrappers/chat_with_website), you need to mount a local folder inside the cluster node, then make it available on Hayhooks Pod as well.
+
+First, ensure that a local folder is mounted correctly on the KinD cluster node at `/data`:
+
+```yaml
+kind: Cluster
+apiVersion: kind.x-k8s.io/v1alpha4
+nodes:
+  - role: control-plane
+    # ...
+    extraMounts:
+      - hostPath: /path/to/local/pipelines/folder
+        containerPath: /data
+```
+
+Next, make `/data` available as a volume and mount it on Hayhooks Pod. To do that, update your previous Pod configuration to the following:
+
+```yaml
+kind: Pod
+apiVersion: v1
+metadata:
+  name: hayhooks
+  labels:
+    app: haystack
+spec:
+  containers:
+    - image: deepset/hayhooks:v1.16.0
+      name: hayhooks
+      imagePullPolicy: IfNotPresent
+      command: ["/bin/sh", "-c"]
+      args:
+        - |
+          pip install trafilatura && \
+          hayhooks run --host 0.0.0.0
+      volumeMounts:
+        - name: local-data
+          mountPath: /mnt/data
+      env:
+        - name: HAYHOOKS_PIPELINES_DIR
+          value: /mnt/data
+        - name: OPENAI_API_KEY
+          valueFrom:
+            secretKeyRef:
+              name: openai-secret
+              key: api-key
+      resources:
+        limits:
+          memory: "512Mi"
+          cpu: "500m"
+        requests:
+          memory: "256Mi"
+          cpu: "250m"
+  volumes:
+    - name: local-data
+      hostPath:
+        path: /data
+        type: Directory
+
+```
+
+Note that:
+
+- We changed the Hayhooks container `command` to install the `trafilatura` dependency before startup, since it's needed for our [chat_with_website](https://github.com/deepset-ai/hayhooks/tree/main/examples/pipeline_wrappers/chat_with_website) example pipeline. For a real production environment, we recommend creating a custom Hayhooks image as described [here](docker.mdx#customizing-the-haystack-docker-image).
+- We make Hayhooks container read `OPENAI_API_KEY` from a Kubernetes Secret.
+
+Before applying this new configuration, create the `openai-secret`:
+
+```yaml
+apiVersion: v1
+kind: Secret
+metadata:
+  name: openai-secret
+type: Opaque
+data:
+  # Replace the placeholder below with the base64 encoded value of your API key
+  # Generate it using: echo -n $OPENAI_API_KEY | base64
+  api-key: YOUR_BASE64_ENCODED_API_KEY_HERE
+```
+
+After applying this, check your Hayhooks Pod logs, and you'll see that the `chat_with_website` pipelines have already been deployed.
+<ClickableImage src="/img/2dbf42dd2db1cb355ee7222d7f8e96c45b611200d83ca289be3456264a854c38-Screenshot_2025-04-16_at_09.19.14.png" alt="Kubernetes Lens interface displaying pod logs with application startup messages and deployed pipeline confirmation" />
+
+## Roll Out Multiple Pods
+
+Haystack pipelines are usually stateless, which is a perfect use case for distributing the requests to multiple pods running the same set of pipelines. Let's convert the single-Pod configuration to an actual Kubernetes `Deployment`:
+
+```yaml
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: haystack-deployment
+spec:
+  replicas: 3
+  selector:
+    matchLabels:
+      app: haystack
+  template:
+    metadata:
+      labels:
+        app: haystack
+    spec:
+      initContainers:
+        - name: install-dependencies
+          image: python:3.12-slim
+          workingDir: /mnt/data
+          command: ["/bin/bash", "-c"]
+          args:
+            - |
+              echo "Installing dependencies..."
+              pip install trafilatura
+              echo "Dependencies installed successfully!"
+              touch /mnt/data/init-complete
+          volumeMounts:
+            - name: local-data
+              mountPath: /mnt/data
+          resources:
+            requests:
+              memory: "64Mi"
+              cpu: "100m"
+            limits:
+              memory: "128Mi"
+              cpu: "250m"
+      containers:
+        - image: deepset/hayhooks:v1.16.0
+          name: hayhooks
+          imagePullPolicy: IfNotPresent
+          command: ["/bin/sh", "-c"]
+          args:
+            - |
+              pip install trafilatura && \
+              hayhooks run --host 0.0.0.0
+          ports:
+            - containerPort: 1416
+              name: http
+          volumeMounts:
+            - name: local-data
+              mountPath: /mnt/data
+          env:
+            - name: HAYHOOKS_PIPELINES_DIR
+              value: /mnt/data
+            - name: OPENAI_API_KEY
+              valueFrom:
+                secretKeyRef:
+                  name: openai-secret
+                  key: api-key
+          resources:
+            requests:
+              memory: "256Mi"
+              cpu: "250m"
+            limits:
+              memory: "512Mi"
+              cpu: "500m"
+      volumes:
+        - name: local-data
+          hostPath:
+            path: /data
+            type: Directory
+
+```
+
+Implementing the above configuration will create three pods. Each pod will run a different instance of Hayhooks, all serving the same example pipeline provided by the mounted volume in the previous example.
+
+<ClickableImage src="/img/f3f0ac4b22a37039f0837c22b0cb8b640937bbb0db4acfcbdf7bd016b545d84a-Screenshot_2025-04-16_at_09.32.07.png" alt="Kubernetes Lens interface showing three haystack-deployment pods in Running status with their resource configurations" />
+
+Note that the `NodePort` you created before will now act as a load balancer and will distribute incoming requests to the three Hayhooks Pods.

docs-website/docs/development/deployment/openshift.mdx

@@ -0,0 +1,73 @@
+---
+title: "OpenShift"
+id: openshift
+slug: "/openshift"
+description: "Learn how to deploy your applications running Haystack pipelines using OpenShift."
+---
+
+# OpenShift
+
+Learn how to deploy your applications running Haystack pipelines using OpenShift.
+
+## Introduction
+
+OpenShift by Red Hat is a platform that helps create and manage applications built on top of Kubernetes. It can be used to build, update, launch, and oversee applications running Haystack pipelines. A [developer sandbox](https://developers.redhat.com/developer-sandbox) is available, ideal for getting familiar with the platform and building prototypes that can be smoothly moved to production using a public cloud, private network, hybrid cloud, or edge computing.
+
+## Prerequisites
+
+The fastest way to deploy a Haystack pipeline is to deploy an OpenShift application that runs Hayhooks. Before starting, make sure to have the following prerequisites:
+
+- Access to an OpenShift project. Follow RedHat's [instructions](https://developers.redhat.com/developer-sandbox) to create one and start experimenting immediately.
+- Hayhooks is installed. Run `pip install hayhooks` and make sure it works by running `hayhooks --version`. Read more about Hayhooks in our [overview](../hayhooks.mdx) or the [official Hayhooks documentation](https://deepset-ai.github.io/hayhooks/).
+- You can optionally install the OpenShift command-line utility `oc`. Follow the [installation instructions](https://docs.openshift.com/container-platform/4.15/cli_reference/openshift_cli/getting-started-cli.html) for your platform and make sure it works by running `oc—h`.
+
+## Creating a Hayhooks Application
+
+In this guide, we’ll be using the `oc` command line, but you can achieve the same by interacting with the user interface offered by the OpenShift console.
+
+1. The first step is to log into your OpenShift account using `oc`. From the top-right corner of your OpenShift console, click on your username and open the menu. Click **Copy login command** and follow the instructions.
+
+2. The console will show you the exact command to run in your terminal to log in. It’s something like the following:  
+   ```
+   oc login --token=<your-token> --server=https://<your-server-url>:6443
+   ```
+
+3. Assuming you already have a project (it’s the case for the developer sandbox), create an application running the Hayhooks Docker image available on Docker Hub:  
+   Note how you can pass environment variables that your application will use at runtime. In this case, we disable Haystack’s internal telemetry and set an OpenAI key that will be used by the pipelines we’ll eventually deploy in Hayhooks.
+   ```
+   oc new-app deepset/hayhooks:v1.16.0 -e HAYSTACK_TELEMETRY_ENABLED=false -e OPENAI_API_KEY=$OPENAI_API_KEY
+   ```
+
+4. To make sure you make the most out of OpenShift's ability to manage the lifecycle of the application, you can set a [liveness probe](https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/):  
+   ```
+   oc set probe deployment/hayhooks --liveness --get-url=http://:1416/status
+   ```
+
+5. Finally, you can expose our Hayhooks instance to the public Internet:  
+   ```
+   oc expose service/hayhooks
+   ```
+
+6. You can get the public address that was assigned to your application by running:  
+
+   ```
+   oc status
+   ```
+
+   In the output, look for something like this:
+
+   ```
+   In project <your-project-name> on server https://<your-server-url>:6443
+
+   http://hayhooks-XXX.openshiftapps.com to pod port 1416-tcp (svc/hayhooks)
+   ```
+
+7. `http://hayhooks-XXX.openshiftapps.com` will be the public URL serving your Hayhooks instance. At this point, you can query Hayhooks status by running:  
+   ```
+   HAYHOOKS_HOST=hayhooks-XXX.openshiftapps.com HAYHOOKS_PORT=80 hayhooks status
+   ```
+
+8. Lastly, deploy your pipeline as usual:  
+   ```
+   HAYHOOKS_HOST=hayhooks-XXX.openshiftapps.com HAYHOOKS_PORT=80 hayhooks pipeline deploy-files -n my_pipeline /path/to/my_pipeline_dir
+   ```

docs-website/docs/development/enabling-gpu-acceleration.mdx

@@ -0,0 +1,43 @@
+---
+title: "Enabling GPU Acceleration"
+id: enabling-gpu-acceleration
+slug: "/enabling-gpu-acceleration"
+description: "Speed up your Haystack application by engaging the GPU."
+---
+
+import ClickableImage from "@site/src/components/ClickableImage";
+
+# Enabling GPU Acceleration
+
+Speed up your Haystack application by engaging the GPU.
+
+The Transformer models used in Haystack are designed to be run on GPU-accelerated hardware. The steps for GPU acceleration setup depend on the environment that you're working in.
+
+Once you have GPU enabled on your machine, you can set the `device` on which a given model for a component is loaded.
+
+For example, to load a model for the `TransformersChatGenerator`, set `device=ComponentDevice.from_single(Device.gpu(id=0))` or `device = ComponentDevice.from_str("cuda:0")` when initializing.
+
+You can find more information on the [Device management](../concepts/device-management.mdx) page.
+
+### Enabling the GPU in Linux
+
+1. Ensure that you have a fitting version of NVIDIA CUDA installed. To learn how to install CUDA, see the [NVIDIA CUDA Guide for Linux](https://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.html).
+
+2. Run the `nvidia-smi`in the command line to check if the GPU is enabled. If the GPU is enabled, the output shows a list of available GPUs and their memory usage:
+   <ClickableImage src="/img/b44c7f4-gpu_enabled_cropped.png" alt="A screenshot of the command output with the name of the GPU device and its memory usage highlighted." />
+
+### Enabling the GPU in Colab
+
+1. In your Colab environment, select **Runtime>Change Runtime type**.
+<ClickableImage src="/img/85079c7-68747470733a2f2f7261772e67697468756275736572636f6e74656e742e636f6d2f646565707365742d61692f686179737461636b2f6d61696e2f646f63732f696d672f636f6c61625f6770755f72756e74696d652e6a7067.jpeg" alt="Google Colab Runtime menu with Change runtime type option highlighted for selecting GPU acceleration" size="large" />
+
+2. Choose **Hardware accelerator>GPU**.
+3. To check if the GPU is enabled, run:
+
+```python python
+%%bash
+
+nvidia-smi
+```
+
+The output should show the GPUs available and their usage.

docs-website/docs/development/external-integrations-development.mdx

@@ -0,0 +1,18 @@
+---
+title: "External Integrations"
+id: external-integrations-development
+slug: "/external-integrations-development"
+description: "External integrations that enable tracing, monitoring, and deploying your pipelines."
+---
+
+# External Integrations
+
+External integrations that enable tracing, monitoring, and deploying your pipelines.
+
+| Name | Description |
+| --- | --- |
+| [Arize Phoenix](https://haystack.deepset.ai/integrations/arize-phoenix) | Trace your pipelines with Arize Phoenix.                 |
+| [Arize AI](https://haystack.deepset.ai/integrations/arize)              | Trace and monitor your pipelines with Arize AI.          |
+| [Burr](https://haystack.deepset.ai/integrations/burr)                   | Build Burr agents using Haystack.                        |
+| [Context AI](https://haystack.deepset.ai/integrations/context-ai)       | Log conversations for analytics by Context.ai.           |
+| [Ray](https://haystack.deepset.ai/integrations/ray)                     | Run and scale your pipelines with in distributed manner. |

docs-website/docs/development/hayhooks.mdx

@@ -0,0 +1,196 @@
+---
+title: "Hayhooks"
+id: hayhooks
+slug: "/hayhooks"
+description: "Hayhooks is a web application you can use to serve Haystack pipelines through HTTP endpoints. This page provides an overview of the main features of Hayhooks."
+---
+
+# Hayhooks
+
+Hayhooks is a web application you can use to serve Haystack pipelines through HTTP endpoints. This page provides an overview of the main features of Hayhooks.
+
+:::info[Hayhooks Documentation]
+
+For comprehensive documentation, including detailed configuration reference, advanced features,
+and examples, see the [official Hayhooks documentation](https://deepset-ai.github.io/hayhooks/).
+
+The source code is available in the [Hayhooks GitHub repository](https://github.com/deepset-ai/hayhooks).
+:::
+
+## Overview
+
+Hayhooks simplifies the deployment of Haystack pipelines as REST APIs. It allows you to:
+
+- Expose Haystack pipelines as HTTP endpoints, including OpenAI-compatible chat endpoints,
+- Customize logic while keeping minimal boilerplate,
+- Deploy pipelines quickly and efficiently.
+
+### Installation
+
+Install Hayhooks using pip:
+
+```shell
+pip install hayhooks
+```
+
+The `hayhooks` package ships both the server and the client component, and the client is capable of starting the server. From a shell, start the server with:
+
+```shell
+$ hayhooks run
+INFO:     Started server process [44782]
+INFO:     Waiting for application startup.
+INFO:     Application startup complete.
+INFO:     Uvicorn running on http://localhost:1416 (Press CTRL+C to quit)
+```
+
+### Check Status
+
+From a different shell, you can query the status of the server with:
+
+```shell
+$ hayhooks status
+Hayhooks server is up and running.
+```
+
+## Configuration
+
+Hayhooks can be configured in three ways:
+
+1. Using an `.env` file in the project root.
+2. Passing environment variables when running the command.
+3. Using command-line arguments with `hayhooks run`.
+
+For a complete list of environment variables including server settings, CORS, SSL, logging, streaming, and Chainlit UI options, see the [Hayhooks environment variables reference](https://deepset-ai.github.io/hayhooks/reference/environment-variables/).
+
+## Running Hayhooks
+
+To start the server:
+
+```shell
+hayhooks run
+```
+
+This will launch Hayhooks at `HAYHOOKS_HOST:HAYHOOKS_PORT`.
+
+## Deploying a Pipeline
+
+### Steps
+
+1. Prepare a pipeline definition (`.yml` file) and a `pipeline_wrapper.py` file.
+2. Deploy the pipeline:
+
+   ```shell
+   hayhooks pipeline deploy-files -n my_pipeline my_pipeline_dir
+   ```
+3. Access the pipeline at `{pipeline_name}/run` endpoint.
+
+### Pipeline Wrapper
+
+A `PipelineWrapper` class is required to wrap the pipeline:
+
+```python
+from pathlib import Path
+from haystack import Pipeline
+from hayhooks import BasePipelineWrapper
+
+
+class PipelineWrapper(BasePipelineWrapper):
+    def setup(self) -> None:
+        pipeline_yaml = (Path(__file__).parent / "pipeline.yml").read_text()
+        self.pipeline = Pipeline.loads(pipeline_yaml)
+
+    def run_api(self, input_text: str) -> str:
+        result = self.pipeline.run({"input": {"text": input_text}})
+        return result["output"]["text"]
+```
+
+## File Uploads
+
+Hayhooks enables handling file uploads in your pipeline wrapper's `run_api` method by including `files: list[UploadFile] | None = None` as an argument.
+
+```python
+def run_api(self, files: list[UploadFile] | None = None) -> str:
+    if files and len(files) > 0:
+        filenames = [f.filename for f in files if f.filename is not None]
+        file_contents = [f.file.read() for f in files]
+        return f"Received files: {', '.join(filenames)}"
+    return "No files received"
+```
+
+Hayhooks automatically processes uploaded files and passes them to the `run_api` method when present. The HTTP request must be a `multipart/form-data` request. For more details on file uploads, including combining files with parameters, see the [official Hayhooks documentation](https://deepset-ai.github.io/hayhooks/features/file-upload-support/).
+
+## Running Pipelines from the CLI
+
+You can execute a pipeline through the command line using the `hayhooks pipeline run` command. Internally, this triggers the `run_api` method of the pipeline wrapper, passing parameters as a JSON payload.
+
+```shell
+hayhooks pipeline run <pipeline_name> --param 'question="Is this recipe vegan?"'
+```
+
+You can also upload files when running a pipeline:
+
+```shell
+hayhooks pipeline run <pipeline_name> --file file.pdf --param 'question="Is this recipe vegan?"'
+```
+
+For the full CLI reference, see the [Hayhooks CLI documentation](https://deepset-ai.github.io/hayhooks/features/cli-commands/).
+
+## MCP Support
+
+Hayhooks supports the [Model Context Protocol (MCP)](https://modelcontextprotocol.io/) and can act as an MCP Server. It automatically lists your deployed pipelines and agents as MCP Tools using Server-Sent Events (SSE) as the transport method. Agents are deployed using the same `PipelineWrapper` mechanism as pipelines.
+
+To start the Hayhooks MCP server, run:
+
+```shell
+hayhooks mcp run
+```
+
+For each deployed pipeline, Hayhooks uses the pipeline wrapper name as the MCP Tool name and generates the tool schema from the `run_api` method arguments. For details on configuring MCP tools, see the [Hayhooks MCP documentation](https://deepset-ai.github.io/hayhooks/features/mcp-support/).
+
+## OpenAI Compatibility
+
+Hayhooks supports OpenAI-compatible endpoints through the `run_chat_completion` method.
+
+```python
+from hayhooks import BasePipelineWrapper, get_last_user_message
+
+
+class PipelineWrapper(BasePipelineWrapper):
+    def run_chat_completion(self, model: str, messages: list, body: dict):
+        question = get_last_user_message(messages)
+        return self.pipeline.run({"query": question})
+```
+
+This makes Hayhooks pipelines compatible with any tool that supports the OpenAI chat completion API, including streaming responses. For details, see the [Hayhooks OpenAI compatibility documentation](https://deepset-ai.github.io/hayhooks/features/openai-compatibility/).
+
+## Running Programmatically
+
+Hayhooks can be embedded in a FastAPI application:
+
+```python
+import uvicorn
+from hayhooks.settings import settings
+from fastapi import Request
+from hayhooks import create_app
+
+# Create the Hayhooks app
+hayhooks = create_app()
+
+
+# Add a custom route
+@hayhooks.get("/custom")
+async def custom_route():
+    return {"message": "Hi, this is a custom route!"}
+
+
+# Add a custom middleware
+@hayhooks.middleware("http")
+async def custom_middleware(request: Request, call_next):
+    response = await call_next(request)
+    response.headers["X-Custom-Header"] = "custom-header-value"
+    return response
+
+
+if __name__ == "__main__":
+    uvicorn.run("app:hayhooks", host=settings.host, port=settings.port)
+```

docs-website/docs/development/logging.mdx

@@ -0,0 +1,105 @@
+---
+title: "Logging"
+id: logging
+slug: "/logging"
+description: "Logging is crucial for monitoring and debugging LLM applications during development as well as in production. Haystack provides different logging solutions out of the box to get you started quickly, depending on your use case."
+---
+
+import ClickableImage from "@site/src/components/ClickableImage";
+
+# Logging
+
+Logging is crucial for monitoring and debugging LLM applications during development as well as in production. Haystack provides different logging solutions out of the box to get you started quickly, depending on your use case.
+
+## Standard Library Logging (default)
+
+Haystack logs through Python’s standard library. This gives you full flexibility and customizability to adjust the log format according to your needs.
+
+### Changing the Log Level
+
+By default, Haystack's logging level is set to `WARNING`. To display more information, you can change it to `INFO`. This way, not only warnings but also information messages are displayed in the console output.
+
+To change the logging level to `INFO`, run:
+
+```python
+import logging
+
+logging.basicConfig(
+    format="%(levelname)s - %(name)s -  %(message)s",
+    level=logging.WARNING,
+)
+logging.getLogger("haystack").setLevel(logging.INFO)
+```
+
+#### Further Configuration
+
+See [Python’s documentation on logging](https://docs.python.org/3/howto/logging.html) for more advanced configuration.
+
+## Real-Time Pipeline Logging
+
+Use Haystack's [`LoggingTracer`](https://github.com/deepset-ai/haystack/blob/main/haystack/tracing/logging_tracer.py) logs to inspect the data that's flowing through your pipeline in real-time.
+
+This feature is particularly helpful during experimentation and prototyping, as you don’t need to set up any tracing backend beforehand.
+
+Here’s how you can enable this tracer. In this example, we are adding color tags (this is optional) to highlight the components' names and inputs:
+
+```python
+import logging
+from haystack import tracing
+from haystack.tracing.logging_tracer import LoggingTracer
+
+logging.basicConfig(
+    format="%(levelname)s - %(name)s -  %(message)s",
+    level=logging.WARNING,
+)
+logging.getLogger("haystack").setLevel(logging.DEBUG)
+
+tracing.tracer.is_content_tracing_enabled = (
+    True  # to enable tracing/logging content (inputs/outputs)
+)
+tracing.enable_tracing(
+    LoggingTracer(
+        tags_color_strings={
+            "haystack.component.input": "\x1b[1;31m",
+            "haystack.component.name": "\x1b[1;34m",
+        },
+    ),
+)
+```
+
+Here’s what the resulting log would look like when a pipeline is run:
+<ClickableImage src="/img/55c3d5c84282d726c95fb3350ec36be49a354edca8a6164f5dffdab7121cec58-image_2.png" alt="Console output showing Haystack pipeline execution with DEBUG level tracing logs including component names, types, and input/output specifications" />
+
+## Structured Logging
+
+Haystack leverages the [structlog library](https://www.structlog.org/en/stable/) to provide structured key-value logs. This provides additional metadata with each log message and is especially useful if you archive your logs with tools like [ELK](https://www.elastic.co/de/elastic-stack), [Grafana](https://grafana.com/oss/agent/?plcmt=footer), or [Datadog](https://www.datadoghq.com/).
+
+If Haystack detects a [structlog installation](https://www.structlog.org/en/stable/) on your system, it will automatically switch to structlog for logging.
+
+### Console Rendering
+
+To make development a more pleasurable experience, Haystack uses [structlog’s `ConsoleRender`](https://www.structlog.org/en/stable/console-output.html) by default to render structured logs as a nicely aligned and colorful output:
+<ClickableImage src="/img/e49a1f2-Screenshot_2024-02-27_at_16.13.51.png" alt="Python code snippet demonstrating basic logging setup with getLogger and a warning level log message output" />
+
+:::tip[Rich Formatting]
+
+Install [_rich_](https://rich.readthedocs.io/en/stable/index.html) to beautify your logs even more!
+:::
+
+### JSON Rendering
+
+We recommend JSON logging when deploying Haystack to production. Haystack will automatically switch to JSON format if it detects no interactive terminal session. If you want to enforce JSON logging:
+
+- Run Haystack with the environment variable `HAYSTACK_LOGGING_USE_JSON` set to `true`.
+- Or, use Python to tell Haystack to log as JSON:
+
+  ```python
+  import haystack.logging
+
+  haystack.logging.configure_logging(use_json=True)
+  ```
+<ClickableImage src="/img/bff93d4-Screenshot_2024-02-27_at_16.15.35.png" alt="Python code snippet showing structured JSON logging configuration with example JSON formatted log output including event, level, and timestamp fields" />
+
+### Disabling Structured Logging
+
+To disable structured logging despite an existing installation of structlog, set the environment variable `HAYSTACK_LOGGING_IGNORE_STRUCTLOG_ENV_VAR` to `true` when running Haystack.

docs-website/docs/development/tracing.mdx

@@ -0,0 +1,56 @@
+---
+title: "Tracing"
+id: tracing
+slug: "/tracing"
+description: "This page explains how to use tracing in Haystack. It lists the tracing backends Haystack supports out of the box and explains how to enable, configure, and disable tracing."
+---
+
+# Tracing
+
+Traces document the flow of requests through your application and are vital for monitoring applications in production. This helps you understand the execution order of your pipeline components and analyze where your pipeline spends the most time.
+
+Instrumented applications typically send traces to a trace collector or a tracing backend. Haystack provides out-of-the-box support for several backends, and you can also quickly implement support for additional providers of your choosing.
+
+## Supported Tracers
+
+| Tracer | Description |
+| --- | --- |
+| [OpenTelemetry](tracing/opentelemetry.mdx) | Send traces to any [OpenTelemetry](https://opentelemetry.io/)-compatible backend using the `OpenTelemetryTracer` or the `OpenTelemetryConnector` component. Includes a Jaeger setup for local development. |
+| [MLflow](tracing/mlflow.mdx) | Capture traces with [MLflow](https://mlflow.org/)'s native Haystack tracing support. |
+| [Datadog](tracing/datadog.mdx) | Trace your pipelines with [Datadog](https://www.datadoghq.com/) using the `DatadogTracer` or the `DatadogConnector` component. |
+| [Langfuse](tracing/langfuse.mdx) | Trace your pipelines with the [Langfuse](https://langfuse.com/) UI using the `LangfuseTracer` or the `LangfuseConnector` component. |
+| [Weights & Biases Weave](tracing/weave.mdx) | Trace and visualize pipeline execution in [Weights & Biases](https://wandb.ai/site/) using the `WeaveTracer` or the `WeaveConnector` component. |
+| [LoggingTracer](tracing/logging-tracer.mdx) | Inspect the data flowing through your pipeline in real time through logs, with no backend setup. |
+| [Custom Tracer](tracing/custom-tracer.mdx) | Connect any tracing backend by implementing the `Tracer` interface. |
+
+## Enabling and Disabling Tracing
+
+Haystack never enables tracing automatically. To enable it, either call `haystack.tracing.enable_tracing(...)` with the tracer of your choice, or add a tracing connector component such as the [`OpenTelemetryConnector`](tracing/opentelemetry.mdx) or the [`DatadogConnector`](tracing/datadog.mdx) to your pipeline.
+
+To disable an enabled tracer:
+
+```python
+from haystack.tracing import disable_tracing
+
+disable_tracing()
+```
+
+## Content Tracing
+
+Haystack also allows you to trace your pipeline components' input and output values. This is useful for investigating your pipeline execution step by step.
+
+By default, this behavior is disabled to prevent sensitive user information from being sent to your tracing backend.
+
+To enable content tracing, there are two options:
+
+- Set the environment variable `HAYSTACK_CONTENT_TRACING_ENABLED` to `true` when running your Haystack application
+
+— or —
+
+- Explicitly enable content tracing in Python:
+
+  ```python
+  from haystack import tracing
+
+  tracing.tracer.is_content_tracing_enabled = True
+  ```

docs-website/docs/development/tracing/custom-tracer.mdx

@@ -0,0 +1,82 @@
+---
+title: "Custom Tracer"
+id: custom-tracer
+slug: "/tracing-custom-tracer"
+description: "Learn how to connect Haystack to a custom tracing backend by implementing the Tracer interface."
+---
+
+# Custom Tracer
+
+Learn how to connect Haystack to a custom tracing backend by implementing the `Tracer` interface.
+
+<div className="key-value-table">
+
+|  |  |
+| --- | --- |
+| **Base classes** | `Tracer` and `Span` |
+| **How to enable** | Implement the `Tracer` interface, then `tracing.enable_tracing(your_tracer)` |
+| **Content tracing** | Optional. Set `HAYSTACK_CONTENT_TRACING_ENABLED` to `true` to trace component inputs and outputs |
+| **Package** | Built into Haystack |
+| **GitHub link** | https://github.com/deepset-ai/haystack/blob/main/haystack/tracing/tracer.py |
+
+</div>
+
+## Overview
+
+If your tracing backend isn't supported out of the box, you can connect it to Haystack by implementing the `Tracer` interface. This gives you full control over how spans are created and how tags are recorded.
+
+## Usage
+
+1. Implement the `Tracer` interface. The following code snippet provides an example using the OpenTelemetry package:
+
+   ```python
+   import contextlib
+   from typing import Optional, Dict, Any, Iterator
+
+   from opentelemetry import trace
+   from opentelemetry.trace import NonRecordingSpan
+
+   from haystack.tracing import Tracer, Span
+   from haystack.tracing import utils as tracing_utils
+   import opentelemetry.trace
+
+   class OpenTelemetrySpan(Span):
+      def __init__(self, span: opentelemetry.trace.Span) -> None:
+          self._span = span
+
+      def set_tag(self, key: str, value: Any) -> None:
+   			 # Tracing backends usually don't support any tag value
+   			 # `coerce_tag_value` forces the value to either be a Python
+   			 # primitive (int, float, boolean, str) or tries to dump it as string.
+          coerced_value = tracing_utils.coerce_tag_value(value)
+          self._span.set_attribute(key, coerced_value)
+
+   class OpenTelemetryTracer(Tracer):
+      def __init__(self, tracer: opentelemetry.trace.Tracer) -> None:
+          self._tracer = tracer
+
+      @contextlib.contextmanager
+      def trace(self, operation_name: str, tags: Optional[Dict[str, Any]] = None) -> Iterator[Span]:
+          with self._tracer.start_as_current_span(operation_name) as span:
+              span = OpenTelemetrySpan(span)
+              if tags:
+                  span.set_tags(tags)
+
+              yield span
+
+      def current_span(self) -> Optional[Span]:
+          current_span = trace.get_current_span()
+          if isinstance(current_span, NonRecordingSpan):
+              return None
+
+          return OpenTelemetrySpan(current_span)
+   ```
+
+2. Tell Haystack to use your custom tracer:
+
+   ```python
+   from haystack import tracing
+
+   haystack_tracer = OpenTelemetryTracer(tracer)
+   tracing.enable_tracing(haystack_tracer)
+   ```

docs-website/docs/development/tracing/datadog.mdx

@@ -0,0 +1,94 @@
+---
+title: "Datadog"
+id: datadog
+slug: "/tracing-datadog"
+description: "Learn how to trace your Haystack pipelines with Datadog."
+---
+
+# Datadog
+
+Learn how to trace your Haystack pipelines with Datadog.
+
+<div className="key-value-table">
+
+|  |  |
+| --- | --- |
+| **Tracer class** | `DatadogTracer` |
+| **How to enable** | Enable the tracer with `tracing.enable_tracing(DatadogTracer(ddtrace.tracer))`, or add the `DatadogConnector` component to your pipeline |
+| **Content tracing** | Set `HAYSTACK_CONTENT_TRACING_ENABLED` to `true` to trace component inputs and outputs |
+| **Package** | `datadog-haystack` |
+| **API reference** | [datadog](/reference/integrations-datadog) |
+| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/datadog |
+
+</div>
+
+## Overview
+
+Trace your Haystack pipelines with [Datadog](https://www.datadoghq.com/) through [Datadog's tracing library `ddtrace`](https://ddtrace.readthedocs.io/en/stable/). Haystack captures detailed information about pipeline runs, like API calls, context data, and prompts, so you can see the complete trace of your pipeline execution in Datadog.
+
+## Installation
+
+Install the `datadog-haystack` package:
+
+```shell
+pip install datadog-haystack
+```
+
+## Prerequisites
+
+1. A way to receive traces, such as a running [Datadog Agent](https://docs.datadoghq.com/agent/). `ddtrace` sends traces to the Datadog Agent at `localhost:8126` by default.
+2. Configure `ddtrace` through the standard mechanisms, for example the `DD_SERVICE`, `DD_ENV`, and `DD_VERSION` environment variables, or by running your application with the `ddtrace-run` command. See the [ddtrace documentation](https://ddtrace.readthedocs.io/en/stable/) for more details.
+
+## Usage
+
+Enable the `DatadogTracer` directly to trace any Haystack pipeline, without adding a component to it. Make sure to set the `HAYSTACK_CONTENT_TRACING_ENABLED` environment variable before importing any Haystack components.
+
+```python
+import os
+
+os.environ["HAYSTACK_CONTENT_TRACING_ENABLED"] = "true"
+
+import ddtrace
+
+from haystack import Pipeline, tracing
+from haystack.components.builders import ChatPromptBuilder
+from haystack.components.generators.chat import OpenAIChatGenerator
+from haystack.dataclasses import ChatMessage
+
+from haystack_integrations.tracing.datadog import DatadogTracer
+
+# Enable the Datadog tracer
+tracing.enable_tracing(DatadogTracer(ddtrace.tracer))
+
+pipe = Pipeline()
+pipe.add_component("prompt_builder", ChatPromptBuilder())
+pipe.add_component("llm", OpenAIChatGenerator())
+pipe.connect("prompt_builder.prompt", "llm.messages")
+
+messages = [
+    ChatMessage.from_system(
+        "Always respond in German even if some input data is in other languages.",
+    ),
+    ChatMessage.from_user("Tell me about {{location}}"),
+]
+
+response = pipe.run(
+    data={
+        "prompt_builder": {
+            "template_variables": {"location": "Berlin"},
+            "template": messages,
+        },
+    },
+)
+print(response["llm"]["replies"][0])
+```
+
+Each pipeline run produces a trace that includes the entire execution context, including prompts, completions, and metadata. You can then view the traces in your Datadog dashboard.
+
+## Alternative: the DatadogConnector component
+
+If you prefer to manage tracing as part of your pipeline definition (for example, so it serializes to YAML), you can add the `DatadogConnector` component instead. It enables the same Datadog tracing as soon as it is initialized.
+
+:::info
+See the [`DatadogConnector` documentation page](../../pipeline-components/connectors/datadogconnector.mdx) for full usage examples, or check out the [integration page](https://haystack.deepset.ai/integrations/datadog).
+:::

docs-website/docs/development/tracing/langfuse.mdx

@@ -0,0 +1,110 @@
+---
+title: "Langfuse"
+id: langfuse
+slug: "/tracing-langfuse"
+description: "Learn how to trace your Haystack pipelines with Langfuse."
+---
+
+import ClickableImage from "@site/src/components/ClickableImage";
+
+# Langfuse
+
+Learn how to trace your Haystack pipelines with Langfuse.
+
+<div className="key-value-table">
+
+|  |  |
+| --- | --- |
+| **Tracer class** | `LangfuseTracer` |
+| **How to enable** | Enable the tracer with `tracing.enable_tracing(LangfuseTracer(langfuse))`, or add the `LangfuseConnector` component to your pipeline |
+| **Content tracing** | Required. Set `HAYSTACK_CONTENT_TRACING_ENABLED` to `true` |
+| **Package** | `langfuse-haystack` |
+| **API reference** | [langfuse](/reference/integrations-langfuse) |
+| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/langfuse |
+
+</div>
+
+## Overview
+
+Trace your Haystack pipelines with the [Langfuse](https://langfuse.com/) UI. Langfuse captures detailed information about pipeline runs, like API calls, context data, prompts, and more. Use it to monitor model performance such as token usage and cost, find areas for improvement, and create datasets from your pipeline executions.
+
+## Installation
+
+Install the `langfuse-haystack` package:
+
+```shell
+pip install langfuse-haystack
+```
+
+## Prerequisites
+
+1. An active Langfuse [account](https://cloud.langfuse.com/).
+2. Set the `LANGFUSE_SECRET_KEY` and `LANGFUSE_PUBLIC_KEY` environment variables with your Langfuse secret and public keys, found in your account profile.
+3. Set the `HAYSTACK_CONTENT_TRACING_ENABLED` environment variable to `true` to enable tracing.
+
+:::info[Usage Notice]
+To ensure proper tracing, always set environment variables before importing any Haystack components. This is crucial because Haystack initializes its internal tracing components during import. An even better practice is to set these environment variables in your shell before running the script.
+:::
+
+## Usage
+
+Enable the `LangfuseTracer` directly to trace any Haystack pipeline, without adding a component to it.
+
+```python
+import os
+
+os.environ["LANGFUSE_HOST"] = "https://cloud.langfuse.com"
+os.environ["LANGFUSE_SECRET_KEY"] = "<your-secret-key>"
+os.environ["LANGFUSE_PUBLIC_KEY"] = "<your-public-key>"
+os.environ["HAYSTACK_CONTENT_TRACING_ENABLED"] = "true"
+
+from langfuse import Langfuse
+
+from haystack import Pipeline, tracing
+from haystack.components.builders import ChatPromptBuilder
+from haystack.components.generators.chat import OpenAIChatGenerator
+from haystack.dataclasses import ChatMessage
+
+from haystack_integrations.tracing.langfuse import LangfuseTracer
+
+# Enable the Langfuse tracer. The client reads your keys from the environment.
+langfuse = Langfuse()
+langfuse_tracer = LangfuseTracer(langfuse, name="Chat example")
+tracing.enable_tracing(langfuse_tracer)
+
+pipe = Pipeline()
+pipe.add_component("prompt_builder", ChatPromptBuilder())
+pipe.add_component("llm", OpenAIChatGenerator())
+pipe.connect("prompt_builder.prompt", "llm.messages")
+
+messages = [
+    ChatMessage.from_system(
+        "Always respond in German even if some input data is in other languages.",
+    ),
+    ChatMessage.from_user("Tell me about {{location}}"),
+]
+
+response = pipe.run(
+    data={
+        "prompt_builder": {
+            "template_variables": {"location": "Berlin"},
+            "template": messages,
+        },
+    },
+)
+print(response["llm"]["replies"][0])
+
+# Flush any pending spans before the program exits
+langfuse_tracer.flush()
+```
+
+Each pipeline run produces one trace that includes the entire execution context, including prompts, completions, and metadata. You can then view the trace in the Langfuse UI.
+<ClickableImage src="/img/11cec4f-langfuse-generation-span.png" alt="Langfuse trace detail view showing generation span with input prompt, output, metadata, latency, and cost information for a language model call" />
+
+## Alternative: the LangfuseConnector component
+
+If you prefer to manage tracing as part of your pipeline definition, you can add the `LangfuseConnector` component instead. It enables the same Langfuse tracing, exposes the `trace_url` as an output, and supports a custom `SpanHandler` for advanced span processing.
+
+:::info
+See the [`LangfuseConnector` documentation page](../../pipeline-components/connectors/langfuseconnector.mdx) for full usage examples and advanced span customization, or read the [blog post](https://haystack.deepset.ai/blog/langfuse-integration) for a complete walkthrough.
+:::

docs-website/docs/development/tracing/logging-tracer.mdx

@@ -0,0 +1,61 @@
+---
+title: "LoggingTracer"
+id: logging-tracer
+slug: "/tracing-logging-tracer"
+description: "Learn how to inspect the data flowing through your Haystack pipelines in real time with the LoggingTracer."
+---
+
+import ClickableImage from "@site/src/components/ClickableImage";
+
+# LoggingTracer
+
+Learn how to inspect the data flowing through your Haystack pipelines in real time with the `LoggingTracer`.
+
+<div className="key-value-table">
+
+|  |  |
+| --- | --- |
+| **Tracer class** | `LoggingTracer` |
+| **How to enable** | `tracing.enable_tracing(LoggingTracer(...))` |
+| **Content tracing** | Required to log inputs and outputs. Set `tracing.tracer.is_content_tracing_enabled = True` |
+| **Package** | Built into Haystack |
+| **GitHub link** | https://github.com/deepset-ai/haystack/blob/main/haystack/tracing/logging_tracer.py |
+
+</div>
+
+## Overview
+
+Use Haystack's [`LoggingTracer`](https://github.com/deepset-ai/haystack/blob/main/haystack/tracing/logging_tracer.py) logs to inspect the data that's flowing through your pipeline in real time.
+
+This feature is particularly helpful during experimentation and prototyping, as you don’t need to set up any tracing backend beforehand.
+
+## Usage
+
+Here’s how you can enable this tracer. In this example, we are adding color tags (this is optional) to highlight the components' names and inputs:
+
+```python
+import logging
+from haystack import tracing
+from haystack.tracing.logging_tracer import LoggingTracer
+
+logging.basicConfig(
+    format="%(levelname)s - %(name)s -  %(message)s",
+    level=logging.WARNING,
+)
+logging.getLogger("haystack").setLevel(logging.DEBUG)
+
+tracing.tracer.is_content_tracing_enabled = (
+    True  # to enable tracing/logging content (inputs/outputs)
+)
+tracing.enable_tracing(
+    LoggingTracer(
+        tags_color_strings={
+            "haystack.component.input": "\x1b[1;31m",
+            "haystack.component.name": "\x1b[1;34m",
+        },
+    ),
+)
+```
+
+Here’s what the resulting log would look like when a pipeline is run:
+<ClickableImage src="/img/55c3d5c84282d726c95fb3350ec36be49a354edca8a6164f5dffdab7121cec58-image_2.png" alt="Console output showing Haystack pipeline execution with DEBUG level tracing logs including component names, types, and input/output specifications" />

docs-website/docs/development/tracing/mlflow.mdx

@@ -0,0 +1,51 @@
+---
+title: "MLflow"
+id: mlflow
+slug: "/tracing-mlflow"
+description: "Learn how to trace your Haystack pipelines with MLflow."
+---
+
+# MLflow
+
+Learn how to trace your Haystack pipelines with MLflow.
+
+<div className="key-value-table">
+
+|  |  |
+| --- | --- |
+| **How to enable** | `mlflow.haystack.autolog()` |
+| **Content tracing** | Captured automatically, including latencies, token usage, cost, and exceptions |
+| **Package** | `mlflow` |
+| **Integration guide** | https://haystack.deepset.ai/integrations/mlflow |
+
+</div>
+
+## Overview
+
+[MLflow](https://mlflow.org/) is an open-source platform for managing the end-to-end machine learning and AI lifecycle. MLflow provides native tracing support for Haystack, so you can capture traces from all your pipelines and components with a single line of code.
+
+## Installation
+
+Install MLflow:
+
+```shell
+pip install mlflow
+```
+
+## Usage
+
+Enable automatic tracing for all Haystack pipelines and components:
+
+```python
+import mlflow
+
+mlflow.haystack.autolog()
+# Optionally set an experiment name
+mlflow.set_experiment("Haystack")
+```
+
+This automatically captures traces from all Haystack pipelines and components, including latencies, token usage, cost, and any exceptions.
+
+:::info
+Check out the [MLflow Haystack integration guide](https://haystack.deepset.ai/integrations/mlflow) for a full walkthrough with examples.
+:::

docs-website/docs/development/tracing/opentelemetry.mdx

@@ -0,0 +1,153 @@
+---
+title: "OpenTelemetry"
+id: opentelemetry
+slug: "/tracing-opentelemetry"
+description: "Learn how to trace your Haystack pipelines with OpenTelemetry."
+---
+
+import ClickableImage from "@site/src/components/ClickableImage";
+
+# OpenTelemetry
+
+Learn how to trace your Haystack pipelines with OpenTelemetry.
+
+<div className="key-value-table">
+
+|  |  |
+| --- | --- |
+| **Tracer class** | `OpenTelemetryTracer` |
+| **How to enable** | Configure an OpenTelemetry `TracerProvider`, then enable the tracer with `tracing.enable_tracing(OpenTelemetryTracer(trace.get_tracer("my_application")))`, or add the `OpenTelemetryConnector` component to your pipeline |
+| **Content tracing** | Set `HAYSTACK_CONTENT_TRACING_ENABLED` to `true` to trace component inputs and outputs |
+| **Package** | `opentelemetry-haystack` |
+| **API reference** | [opentelemetry](/reference/integrations-opentelemetry) |
+| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/opentelemetry |
+
+</div>
+
+## Overview
+
+[OpenTelemetry](https://opentelemetry.io/) is an open-source observability framework for collecting traces, metrics, and logs. Haystack integrates with OpenTelemetry, so you can send traces of your pipeline runs to any OpenTelemetry-compatible backend.
+
+:::info[Moving to an integration]
+`OpenTelemetryTracer` is deprecated in Haystack core and is moving to the `opentelemetry-haystack` package. Starting with Haystack 3.0, OpenTelemetry tracing is no longer auto-enabled when `opentelemetry-sdk` is installed. Install the integration and either enable the `OpenTelemetryTracer` directly or add the `OpenTelemetryConnector` component to your pipeline.
+:::
+
+## Installation
+
+Install the `opentelemetry-haystack` package:
+
+```shell
+pip install opentelemetry-haystack
+```
+
+To add traces to even deeper levels of your pipelines, we recommend you check out [OpenTelemetry integrations](https://opentelemetry.io/ecosystem/registry/?s=python), such as:
+
+- [`urllib3` instrumentation](https://github.com/open-telemetry/opentelemetry-python-contrib/tree/main/instrumentation/opentelemetry-instrumentation-urllib3) for tracing HTTP requests in your pipeline,
+- [OpenAI instrumentation](https://github.com/traceloop/openllmetry/tree/main/packages/opentelemetry-instrumentation-openai) for tracing OpenAI requests.
+
+## Prerequisites
+
+A configured OpenTelemetry `TracerProvider` with an exporter, for example an OTLP exporter that sends traces to a collector or a backend. Set up the provider before enabling the tracer.
+
+## Usage
+
+Enable the `OpenTelemetryTracer` directly to trace any Haystack pipeline, without adding a component to it. Configure your `TracerProvider` and set the `HAYSTACK_CONTENT_TRACING_ENABLED` environment variable before importing any Haystack components.
+
+```python
+import os
+
+os.environ["HAYSTACK_CONTENT_TRACING_ENABLED"] = "true"
+
+from opentelemetry import trace
+from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
+from opentelemetry.sdk.resources import Resource
+from opentelemetry.sdk.trace import TracerProvider
+from opentelemetry.sdk.trace.export import BatchSpanProcessor
+from opentelemetry.semconv.resource import ResourceAttributes
+
+# Configure the OpenTelemetry SDK. A service name is required for most backends.
+resource = Resource(attributes={ResourceAttributes.SERVICE_NAME: "haystack"})
+tracer_provider = TracerProvider(resource=resource)
+tracer_provider.add_span_processor(
+    BatchSpanProcessor(OTLPSpanExporter(endpoint="http://localhost:4318/v1/traces")),
+)
+trace.set_tracer_provider(tracer_provider)
+
+from haystack import tracing
+from haystack_integrations.tracing.opentelemetry import OpenTelemetryTracer
+
+# Enable the OpenTelemetry tracer
+tracing.enable_tracing(OpenTelemetryTracer(trace.get_tracer("my_application")))
+```
+
+Each pipeline run then produces a trace that includes the entire execution context, including prompts, completions, and metadata. You can view the traces in your OpenTelemetry-compatible backend.
+
+## Alternative: the OpenTelemetryConnector component
+
+If you prefer to manage tracing as part of your pipeline definition, you can add the `OpenTelemetryConnector` component instead. It enables the same OpenTelemetry tracing as soon as it is initialized.
+
+:::info
+See the [`OpenTelemetryConnector` documentation page](../../pipeline-components/connectors/opentelemetryconnector.mdx) for full usage examples, or check out the [integration page](https://haystack.deepset.ai/integrations/opentelemetry).
+:::
+
+## Visualizing Traces During Development
+
+Use [Jaeger](https://www.jaegertracing.io/docs/1.6/getting-started/) as a lightweight tracing backend for local pipeline development. This allows you to experiment with tracing without the need for a complex tracing backend.
+<ClickableImage src="/img/dd906d7-Screenshot_2024-02-22_at_16.51.01.png" alt="Jaeger UI trace timeline displaying haystack pipeline execution with component spans showing duration and nesting of operations" />
+
+1. Run the Jaeger container. This creates a tracing backend as well as a UI to visualize the traces:
+
+   ```shell
+   docker run --rm -d --name jaeger \
+     -e COLLECTOR_ZIPKIN_HOST_PORT=:9411 \
+     -p 6831:6831/udp \
+     -p 6832:6832/udp \
+     -p 5778:5778 \
+     -p 16686:16686 \
+     -p 4317:4317 \
+     -p 4318:4318 \
+     -p 14250:14250 \
+     -p 14268:14268 \
+     -p 14269:14269 \
+     -p 9411:9411 \
+     jaegertracing/all-in-one:latest
+   ```
+2. Install the integration and the OTLP exporter:
+
+   ```shell
+   pip install opentelemetry-haystack
+   pip install opentelemetry-exporter-otlp
+   ```
+3. Configure `OpenTelemetry` to use the Jaeger backend and enable the tracer:
+
+   ```python
+   from opentelemetry import trace
+   from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
+   from opentelemetry.sdk.resources import Resource
+   from opentelemetry.sdk.trace import TracerProvider
+   from opentelemetry.sdk.trace.export import BatchSpanProcessor
+   from opentelemetry.semconv.resource import ResourceAttributes
+
+   from haystack import tracing
+   from haystack_integrations.tracing.opentelemetry import OpenTelemetryTracer
+
+   # Service name is required for most backends
+   resource = Resource(attributes={
+       ResourceAttributes.SERVICE_NAME: "haystack"
+   })
+
+   tracer_provider = TracerProvider(resource=resource)
+   processor = BatchSpanProcessor(OTLPSpanExporter(endpoint="http://localhost:4318/v1/traces"))
+   tracer_provider.add_span_processor(processor)
+   trace.set_tracer_provider(tracer_provider)
+
+   tracing.enable_tracing(OpenTelemetryTracer(trace.get_tracer("my_application")))
+   ```
+4. Run your pipeline:
+
+   ```python
+   ...
+   pipeline.run(...)
+   ...
+   ```
+5. Inspect the traces in the UI provided by Jaeger at [http://localhost:16686](http://localhost:16686/search).

docs-website/docs/development/tracing/weave.mdx

@@ -0,0 +1,93 @@
+---
+title: "Weights & Biases Weave"
+id: weave
+slug: "/tracing-weave"
+description: "Learn how to trace your Haystack pipelines with Weights & Biases Weave."
+---
+
+# Weights & Biases Weave
+
+Learn how to trace your Haystack pipelines with Weights & Biases Weave.
+
+<div className="key-value-table">
+
+|  |  |
+| --- | --- |
+| **Tracer class** | `WeaveTracer` |
+| **How to enable** | Enable the tracer with `tracing.enable_tracing(WeaveTracer(project_name="..."))`, or add the `WeaveConnector` component to your pipeline |
+| **Content tracing** | Required. Set `HAYSTACK_CONTENT_TRACING_ENABLED` to `true` |
+| **Package** | `weave-haystack` |
+| **API reference** | [Weave](/reference/integrations-weave) |
+| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/weave |
+
+</div>
+
+## Overview
+
+Trace and visualize your pipeline execution in [Weights & Biases](https://wandb.ai/site/). Information captured by the Haystack tracing tool, such as API calls, context data, and prompts, is sent to Weights & Biases, where you can see the complete trace of your pipeline execution.
+
+## Installation
+
+Install the `weave-haystack` package:
+
+```shell
+pip install weave-haystack
+```
+
+## Prerequisites
+
+1. A Weave account. You can sign up for free on the [Weights & Biases website](https://wandb.ai/site).
+2. Set the `WANDB_API_KEY` environment variable with your Weights & Biases API key. Once logged in, you can find your API key on [your home page](https://wandb.ai/home).
+3. Set the `HAYSTACK_CONTENT_TRACING_ENABLED` environment variable to `true`.
+
+## Usage
+
+Enable the `WeaveTracer` directly to trace any Haystack pipeline, without adding a component to it. The `project_name` is the name that will appear in your Weave project.
+
+```python
+import os
+
+os.environ["HAYSTACK_CONTENT_TRACING_ENABLED"] = "true"
+
+from haystack import Pipeline, tracing
+from haystack.components.builders import ChatPromptBuilder
+from haystack.components.generators.chat import OpenAIChatGenerator
+from haystack.dataclasses import ChatMessage
+
+from haystack_integrations.tracing.weave import WeaveTracer
+
+# Enable the Weave tracer
+tracing.enable_tracing(WeaveTracer(project_name="test_pipeline"))
+
+pipe = Pipeline()
+pipe.add_component("prompt_builder", ChatPromptBuilder())
+pipe.add_component("llm", OpenAIChatGenerator())
+pipe.connect("prompt_builder.prompt", "llm.messages")
+
+messages = [
+    ChatMessage.from_system(
+        "Always respond in German even if some input data is in other languages.",
+    ),
+    ChatMessage.from_user("Tell me about {{location}}"),
+]
+
+response = pipe.run(
+    data={
+        "prompt_builder": {
+            "template_variables": {"location": "Berlin"},
+            "template": messages,
+        },
+    },
+)
+print(response["llm"]["replies"][0])
+```
+
+You can then see the complete trace for your pipeline at `https://wandb.ai/<user_name>/projects` under the project name you specified.
+
+## Alternative: the WeaveConnector component
+
+If you prefer to manage tracing as part of your pipeline definition, you can add the `WeaveConnector` component instead. It enables the same Weave tracing as soon as it runs.
+
+:::info
+See the [`WeaveConnector` documentation page](../../pipeline-components/connectors/weaveconnector.mdx) for full usage examples.
+:::