deepset-ai--haystack/docs-website/docs/concepts/pipelines/serialization.mdx

---
title: "Serializing Pipelines"
id: serialization
slug: "/serialization"
description: "Save your pipelines into a custom format and explore the serialization options."
---

# Serializing Pipelines

Save your pipelines into a custom format and explore the serialization options.

Serialization means converting a pipeline to a format that you can save on your disk and load later.

Haystack supports YAML format for pipeline serialization.

## Converting a Pipeline to YAML

Use the `dumps()` method to convert a Pipeline object to YAML:

```python
from haystack import Pipeline

pipe = Pipeline()
print(pipe.dumps())

# Prints:
#
# components: {}
# connections: []
# max_runs_per_component: 100
# metadata: {}
```

You can also use `dump()` method to save the YAML representation of a pipeline in a file:

```python
with open("/content/test.yml", "w") as file:
    pipe.dump(file)
```

## Converting a Pipeline Back to Python

You can convert a YAML pipeline back into Python. Use the `loads()` method to convert a string representation of a pipeline (`str`, `bytes` or `bytearray`)  or the `load()` method to convert a pipeline represented in a file-like object into a corresponding Python object.

Both loading methods support callbacks that let you modify components during the deserialization process. Deserialization is gated by a trusted-module allowlist, so pipelines referencing classes outside of it fail to load until you extend the allowlist — see [Deserialization Security](#deserialization-security) below.

Here is an example script:

```python
from haystack import Pipeline
from haystack.core.serialization import DeserializationCallbacks
from typing import Type, Dict, Any

# This is the YAML you want to convert to Python:
pipeline_yaml = """
components:
  cleaner:
    init_parameters:
      remove_empty_lines: true
      remove_extra_whitespaces: true
      remove_regex: null
      remove_repeated_substrings: false
      remove_substrings: null
    type: haystack.components.preprocessors.document_cleaner.DocumentCleaner
  converter:
    init_parameters:
      encoding: utf-8
    type: haystack.components.converters.txt.TextFileToDocument
connections:
- receiver: cleaner.documents
  sender: converter.documents
max_runs_per_component: 100
metadata: {}
"""


def component_pre_init_callback(
    component_name: str,
    component_cls: Type,
    init_params: Dict[str, Any],
):
    # This function gets called every time a component is deserialized.
    if component_name == "cleaner":
        assert "DocumentCleaner" in component_cls.__name__
        # Modify the init parameters. The modified parameters are passed to
        # the init method of the component during deserialization.
        init_params["remove_empty_lines"] = False
        print("Modified 'remove_empty_lines' to False in 'cleaner' component")
    else:
        print(f"Not modifying component {component_name} of class {component_cls}")


pipe = Pipeline.loads(
    pipeline_yaml,
    callbacks=DeserializationCallbacks(component_pre_init_callback),
)
```

## Deserialization Security

Loading a pipeline instantiates the classes referenced in the serialized data. To prevent a crafted YAML file from importing and instantiating arbitrary classes, `Pipeline.load`, `Pipeline.loads`, and `Pipeline.from_dict` refuse to import classes from modules outside a trusted-module allowlist and raise a `DeserializationError` instead.

By default, the allowlist contains `haystack`, `haystack_integrations`, `haystack_experimental`, `builtins`, `typing`, and `collections`. Dangerous builtins such as `eval`, `exec`, `compile`, `__import__`, `open`, and `getattr` are blocked even though `builtins` is allowlisted.

### Allowing Custom Modules

Pipelines that reference custom components or callables in other packages fail to load until you add the modules to the allowlist. You can extend it in three ways:

```python
from haystack import Pipeline

# 1. Per call: pass additional module patterns for this deserialization only
pipe = Pipeline.load(open("pipeline.yaml"), allowed_modules=["mypkg.*"])

# 2. Process-wide: extend the allowlist programmatically
from haystack.core.serialization import allow_deserialization_module

allow_deserialization_module("mypkg")
```

```shell
# 3. Environment variable with comma-separated patterns, read on every deserialization call
export HAYSTACK_DESERIALIZATION_ALLOWLIST="mypkg.*,otherpkg.*"
```

Patterns are matched as prefixes by default (`"mypkg"` matches `mypkg` and any of its submodules), or as `fnmatch` globs if they contain `*`, `?`, or `[` somewhere other than a trailing `.*`. A trailing `.*` is treated as a prefix match, so `"mypkg"` and `"mypkg.*"` behave identically.

If the source of the serialized data is fully trusted, you can bypass the allowlist entirely with `unsafe=True`:

```python
pipe = Pipeline.load(open("pipeline.yaml"), unsafe=True)
```

Only use `unsafe=True` when you fully trust where the serialized pipeline comes from — it also lifts the block on dangerous builtins.

### Nested Init Parameter Validation

As an additional safeguard, deserialization validates the keys of `init_parameters` against the class's `__init__` signature before recursing into any nested `{"type": "...", "init_parameters": {...}}` dictionary. A nested dictionary whose key is not an accepted parameter name is rejected with a `DeserializationError` *before* the nested type is imported, which blocks attempts to smuggle untrusted classes into unused parameter slots. Classes whose constructor takes `**kwargs` are exempt, since their accepted parameter set cannot be statically determined.

This validation may surface pre-existing bugs in YAML files — for example typos, leftovers from renamed or removed parameters, or stale snapshots from older Haystack versions. The fix is to update the YAML so each nested-component key matches a real `__init__` parameter of the parent class.

## Default Serialization Behavior

The serialization system uses `default_to_dict` and `default_from_dict` to handle many object types automatically. You typically do **not** need to implement custom `to_dict`/`from_dict` for:

- **Secrets**: serialized and deserialized automatically so that sensitive values aren't stored in plain text.
- **ComponentDevice**: device configuration is detected and restored automatically.
- **Objects with their own `to_dict`/`from_dict`**: any init parameter whose type defines `to_dict()` is serialized by calling it; any dict in `init_parameters` with a `type` key pointing to a class with `from_dict()` is deserialized automatically.

To serialize or deserialize a single component, you can use `component_to_dict` and `component_from_dict` from `haystack.core.serialization`. They use the default behavior above as a fallback when the component doesn't define custom `to_dict`/`from_dict`:

```python
from haystack import component
from haystack.core.serialization import component_from_dict, component_to_dict


@component
class Greeter:
    def __init__(self, message: str = "Hello"):
        self.message = message

    @component.output_types(greeting=str)
    def run(self, name: str):
        return {"greeting": f"{self.message}, {name}!"}


# Serialize a component instance to a dictionary
greeter = Greeter(message="Hi")
data = component_to_dict(greeter, "my_greeter")

# Deserialize back to a component instance
restored = component_from_dict(Greeter, data, "my_greeter")
assert restored.message == greeter.message
```

:::caution[Init parameters must be stored as instance attributes]

Default serialization only works when there is a **1:1 mapping** between init parameter names and instance attributes. For every argument in `__init__`, the component must assign it to an attribute with the same name. For example, if you have `def __init__(self, prompt: str)`, you must have `self.prompt = prompt` in the class. Otherwise the serialization logic can't find the value to serialize and raises an error or uses the default value if the parameter has one.
:::

## Performing Custom Serialization

Pipelines and components in Haystack can serialize simple components, including custom ones, out of the box. Code like this just works:

```python
from haystack import component


@component
class RepeatWordComponent:
    def __init__(self, times: int):
        self.times = times

    @component.output_types(result=str)
    def run(self, word: str):
        return word * self.times
```

On the other hand, this code doesn't work if the final format is JSON, as the `set` type is not JSON-serializable:

```python
from haystack import component


@component
class SetIntersector:
    def __init__(self, intersect_with: set):
        self.intersect_with = intersect_with

    @component.output_types(result=set)
    def run(self, data: set):
        return data.intersection(self.intersect_with)
```

In such cases, you can provide your own implementation  `from_dict` and `to_dict` to components:

```python
from haystack import component, default_from_dict, default_to_dict


class SetIntersector:
    def __init__(self, intersect_with: set):
        self.intersect_with = intersect_with

    @component.output_types(result=set)
    def run(self, data: set):
        return data.intersect(self.intersect_with)

    def to_dict(self):
        return default_to_dict(self, intersect_with=list(self.intersect_with))

    @classmethod
    def from_dict(cls, data):
        # convert the set into a list for the dict representation,
        # so it can be converted to JSON
        data["intersect_with"] = set(data["intersect_with"])
        return default_from_dict(cls, data)
```

## Saving a Pipeline to a Custom Format

Once a pipeline is available in its dictionary format, the last step of serialization is to convert that dictionary into a format you can store or send over the wire. Haystack supports YAML out of the box, but if you need a different format, you can write a custom Marshaller.

A `Marshaller` is a Python class responsible for converting text to a dictionary and a dictionary to text according to a certain format. Marshallers must respect the `Marshaller` [protocol](https://github.com/deepset-ai/haystack/blob/main/haystack/marshal/protocol.py), providing the methods `marshal` and `unmarshal`.

This is the code for a custom TOML marshaller that relies on the `rtoml` library:

```python
# This code requires a `pip install rtoml`
from typing import Dict, Any, Union
import rtoml


class TomlMarshaller:
    def marshal(self, dict_: Dict[str, Any]) -> str:
        return rtoml.dumps(dict_)

    def unmarshal(self, data_: Union[str, bytes]) -> Dict[str, Any]:
        return dict(rtoml.loads(data_))
```

You can then pass a Marshaller instance to the methods `dump`, `dumps`, `load`, and `loads`:

```python
from haystack import Pipeline
from my_custom_marshallers import TomlMarshaller

pipe = Pipeline()
pipe.dumps(TomlMarshaller())
# prints:
# 'max_runs_per_component = 100\nconnections = []\n\n[metadata]\n\n[components]\n'
```

## Additional References

:notebook: Tutorial: [Serializing LLM Pipelines](https://haystack.deepset.ai/tutorials/29_serializing_pipelines)