chore: import upstream snapshot with attribution

test/core/pipeline/features/README.md

@@ -0,0 +1,135 @@
+# `Pipeline.run()` behavioural tests
+
+This module contains all behavioural tests for `Pipeline.run()`.
+
+`pipeline_run.feature` contains the definition of the tests using a subset of the [Gherkin language](https://cucumber.io/docs/gherkin/). It's not the full language because we're using `pytest-bdd` and it doesn't implement it in full, but it's good enough for our use case. For more info see the [project `README.md`](https://github.com/pytest-dev/pytest-bdd).
+
+There are two cases covered by these tests:
+
+1. `Pipeline.run()` returns some output
+2. `Pipeline.run()` raises an exception
+
+### Correct Pipeline
+
+In the first case to add a new test you need add a new entry in the `Examples` of the `Running a correct Pipeline` scenario outline and create the corresponding step that creates the `Pipeline` you need to test.
+
+For example to add a test for a linear `Pipeline` I add a new `that is linear` kind in `pipeline_run.feature`.
+
+```gherkin
+    Scenario Outline: Running a correct Pipeline
+        Given a pipeline <kind>
+        When I run the Pipeline
+        Then it should return the expected result
+
+        Examples:
+        | kind |
+        | that has no components |
+        | that is linear |
+```
+
+Then define a new `pipeline_that_is_linear` function in `test_run.py`.
+The function must be decorated with `@given` and return a tuple containing the `Pipeline` instance and a list of `PipelineRunData` instances.
+`PipelineRunData` is a dataclass that stores all the information necessary to verify the `Pipeline` ran as expected.
+The `@given` arguments must be the full step name, `"a pipeline that is linear"` in this case, and `target_fixture` must be set to `"pipeline_data"`.
+
+```python
+@given("a pipeline that is linear", target_fixture="pipeline_data")
+def pipeline_that_is_linear():
+    pipeline = Pipeline()
+    pipeline.add_component("first_addition", AddFixedValue(add=2))
+    pipeline.add_component("second_addition", AddFixedValue())
+    pipeline.add_component("double", Double())
+    pipeline.connect("first_addition", "double")
+    pipeline.connect("double", "second_addition")
+
+    return (
+        pipeline,
+        [
+            PipelineRunData(
+                inputs={"first_addition": {"value": 1}},
+                expected_outputs={"second_addition": {"result": 7}},
+                expected_run_order=["first_addition", "double", "second_addition"],
+            )
+        ],
+    )
+```
+
+Some kinds of `Pipeline`s require multiple runs to verify they work correctly, for example those with multiple branches.
+For this reason we can return a list of `PipelineRunData`, we'll run the `Pipeline` for each instance.
+For example, we could test two different runs of the same pipeline like this:
+
+```python
+@given("a pipeline that is linear", target_fixture="pipeline_data")
+def pipeline_that_is_linear():
+    pipeline = Pipeline()
+    pipeline.add_component("first_addition", AddFixedValue(add=2))
+    pipeline.add_component("second_addition", AddFixedValue())
+    pipeline.add_component("double", Double())
+    pipeline.connect("first_addition", "double")
+    pipeline.connect("double", "second_addition")
+
+    return (
+        pipeline,
+        [
+            PipelineRunData(
+                inputs={"first_addition": {"value": 1}},
+                include_outputs_from=set(),
+                expected_outputs={"second_addition": {"result": 7}},
+                expected_run_order=["first_addition", "double", "second_addition"],
+            ),
+            PipelineRunData(
+                inputs={"first_addition": {"value": 100}},
+                include_outputs_from=set(),
+                expected_outputs={"first_addition": {"value": 206}},
+                expected_run_order=["first_addition", "double", "second_addition"],
+            ),
+        ],
+    )
+```
+
+### Bad Pipeline
+
+The second case is similar to the first one.
+In this case we test that a `Pipeline` with an infinite loop raises `PipelineMaxLoops`.
+
+```gherkin
+    Scenario Outline: Running a bad Pipeline
+        Given a pipeline <kind>
+        When I run the Pipeline
+        Then it must have raised <exception>
+
+        Examples:
+        | kind | exception |
+        | that has an infinite loop | PipelineMaxLoops |
+```
+
+In a similar way as first case we need to defined a new `pipeline_that_has_an_infinite_loop` function in `test_run.py`, with some small differences.
+The only difference from the first case is the last value returned by the function, we just omit the expected outputs and the expected run order.
+
+```python
+@given("a pipeline that has an infinite loop", target_fixture="pipeline_data")
+def pipeline_that_has_an_infinite_loop():
+    def custom_init(self):
+        component.set_input_type(self, "x", int)
+        component.set_input_type(self, "y", int, 1)
+        component.set_output_types(self, a=int, b=int)
+
+    FakeComponent = component_class("FakeComponent", output={"a": 1, "b": 1}, extra_fields={"__init__": custom_init})
+    pipe.add_component("first", FakeComponent())
+    pipe.add_component("second", FakeComponent())
+    pipe.connect("first.a", "second.x")
+    pipe.connect("second.b", "first.y")
+    return pipe, [PipelineRunData({"first": {"x": 1}})]
+```
+
+## Why?
+
+As the time of writing, tests that invoke `Pipeline.run()` are scattered between different files with very little clarity on what they are intended to test - the only indicators are the name of each test itself and the name of their parent module. This makes it difficult to understand which behaviours are being tested, if they are tested redundantly or if they work correctly.
+
+The introduction of the Gherkin file allows for a single "source of truth" that enumerates (ideally, in an exhaustive manner) all the behaviours of the pipeline execution logic that we wish to test. This intermediate mapping of behaviours to actual test cases is meant to provide an overview of the latter and reduce the cognitive overhead of understanding them. When writing new tests, we now "tag" them with a specific behavioural parameter that's specified in a Gherkin scenario.
+
+This tag and behavioural parameter mapping is meant to be 1 to 1, meaning each "Given" step must map to one and only one function. If multiple function are marked with `@given("step name")` the last declaration will override all the previous ones. So it's important to verify that there are no other existing steps with the same name when adding a new one.
+
+While one could functionally do the same with well-defined test names and detailed comments on what is being tested, it would still lack the overview that the above approach provides. It's also extensible in that new scenarios with different behaviours can be introduced easily (e.g: for `async` pipeline execution logic).
+
+Apart from the above, the newly introduced harness ensures that all behavioural pipeline tests return a structured result, which simplifies checking of side-effects.

test/core/pipeline/features/conftest.py

@@ -0,0 +1,188 @@
+# SPDX-FileCopyrightText: 2022-present deepset GmbH <info@deepset.ai>
+#
+# SPDX-License-Identifier: Apache-2.0
+
+import asyncio
+from dataclasses import dataclass, field
+from typing import Any
+
+import pytest
+from pandas import DataFrame
+from pytest_bdd import parsers, then, when
+
+from haystack import Pipeline, component
+from test.tracing.utils import SpyingTracer
+
+
+@pytest.fixture(params=["sync", "async"])
+def pipeline_run_mode(request):
+    """
+    Parametrizes each scenario so it runs once through `Pipeline.run` (sync) and once through
+    `Pipeline.run_async` (async), exercising both execution engines.
+    """
+    return request.param
+
+
+@dataclass
+class PipelineRunData:
+    """
+    Holds the inputs and expected outputs for a single Pipeline run.
+    """
+
+    inputs: dict[str, Any]
+    include_outputs_from: set[str] = field(default_factory=set)
+    expected_outputs: dict[str, Any] = field(default_factory=dict)
+    expected_component_calls: dict[tuple[str, int], dict[str, Any]] = field(default_factory=dict)
+
+
+@dataclass
+class _PipelineResult:
+    """
+    Holds the outputs and the run order of a single Pipeline run.
+    """
+
+    outputs: dict[str, Any]
+    component_calls: dict[tuple[str, int], dict[str, Any]] = field(default_factory=dict)
+
+
+@when("I run the Pipeline", target_fixture="pipeline_result")
+def run_pipeline(
+    pipeline_data: tuple[Pipeline, list[PipelineRunData]], spying_tracer: SpyingTracer, pipeline_run_mode: str
+) -> list[tuple[_PipelineResult, PipelineRunData]] | Exception:
+    if pipeline_run_mode == "async":
+        return run_async_pipeline(pipeline_data, spying_tracer)
+    return run_sync_pipeline(pipeline_data, spying_tracer)
+
+
+def run_async_pipeline(
+    pipeline_data: tuple[Pipeline, list[PipelineRunData]], spying_tracer: SpyingTracer
+) -> list[tuple[_PipelineResult, PipelineRunData]] | Exception:
+    """
+    Attempts to run a pipeline with the given inputs.
+    `pipeline_data` is a tuple that must contain:
+    * A Pipeline instance
+    * The data to run the pipeline with
+
+    If successful returns a tuple of the run outputs and the expected outputs.
+    In case an exceptions is raised returns that.
+    """
+    pipeline, pipeline_run_data = pipeline_data[0], pipeline_data[1]
+
+    results: list[_PipelineResult] = []
+
+    async def run_inner(data, include_outputs_from):
+        """Wrapper function to call pipeline.run_async method with required params."""
+        return await pipeline.run_async(data=data.inputs, include_outputs_from=include_outputs_from)
+
+    for data in pipeline_run_data:
+        try:
+            outputs = asyncio.run(run_inner(data, data.include_outputs_from))
+
+            component_calls = {
+                (span.tags["haystack.component.name"], span.tags["haystack.component.visits"]): span.tags[
+                    "haystack.component.input"
+                ]
+                for span in spying_tracer.spans
+                if "haystack.component.name" in span.tags and "haystack.component.visits" in span.tags
+            }
+            results.append(_PipelineResult(outputs=outputs, component_calls=component_calls))
+            spying_tracer.spans.clear()
+        except Exception as e:
+            return e
+
+    return list(zip(results, pipeline_run_data, strict=True))
+
+
+def run_sync_pipeline(
+    pipeline_data: tuple[Pipeline, list[PipelineRunData]], spying_tracer: SpyingTracer
+) -> list[tuple[_PipelineResult, PipelineRunData]] | Exception:
+    """
+    Attempts to run a pipeline with the given inputs.
+    `pipeline_data` is a tuple that must contain:
+    * A Pipeline instance
+    * The data to run the pipeline with
+
+    If successful returns a tuple of the run outputs and the expected outputs.
+    In case an exceptions is raised returns that.
+    """
+    pipeline, pipeline_run_data = pipeline_data[0], pipeline_data[1]
+
+    results: list[_PipelineResult] = []
+
+    for data in pipeline_run_data:
+        try:
+            outputs = pipeline.run(data=data.inputs, include_outputs_from=data.include_outputs_from)
+
+            component_calls = {
+                (span.tags["haystack.component.name"], span.tags["haystack.component.visits"]): span.tags[
+                    "haystack.component.input"
+                ]
+                for span in spying_tracer.spans
+                if "haystack.component.name" in span.tags and "haystack.component.visits" in span.tags
+            }
+            results.append(_PipelineResult(outputs=outputs, component_calls=component_calls))
+            spying_tracer.spans.clear()
+        except Exception as e:
+            return e
+    return list(zip(results, pipeline_run_data, strict=True))
+
+
+@then("it should return the expected result")
+def check_pipeline_result(pipeline_result: list[tuple[_PipelineResult, PipelineRunData]]) -> None:
+    for res, data in pipeline_result:
+        compare_outputs_with_dataframes(res.outputs, data.expected_outputs)
+
+
+@then("components are called with the expected inputs")
+def check_component_calls(pipeline_result: list[tuple[_PipelineResult, PipelineRunData]]) -> None:
+    for res, data in pipeline_result:
+        assert compare_outputs_with_dataframes(res.component_calls, data.expected_component_calls)
+
+
+@then(parsers.parse("it must have raised {exception_class_name}"))
+def check_pipeline_raised(pipeline_result: Exception, exception_class_name: str) -> None:
+    assert pipeline_result.__class__.__name__ == exception_class_name
+
+
+def compare_outputs_with_dataframes(actual: dict, expected: dict) -> bool:
+    """
+    Compare two component_calls or pipeline outputs dictionaries where values may contain DataFrames.
+    """
+    assert actual.keys() == expected.keys()
+
+    for key in actual:
+        actual_data = actual[key]
+        expected_data = expected[key]
+
+        assert actual_data.keys() == expected_data.keys()
+
+        for data_key in actual_data:
+            actual_value = actual_data[data_key]
+            expected_value = expected_data[data_key]
+
+            if isinstance(actual_value, DataFrame) and isinstance(expected_value, DataFrame):
+                assert actual_value.equals(expected_value)
+            else:
+                # We do expected_value first so ANY can be used in expected outputs
+                assert expected_value == actual_value
+
+    return True
+
+
+@component
+class FixedGenerator:
+    def __init__(self, replies):
+        self.replies = replies
+        self.idx = 0
+
+    @component.output_types(replies=list[str])
+    def run(self, prompt: str) -> dict[str, Any]:
+        if self.idx < len(self.replies):
+            replies = [self.replies[self.idx]]
+            self.idx += 1
+        else:
+            self.idx = 0
+            replies = [self.replies[self.idx]]
+            self.idx += 1
+
+        return {"replies": replies}

test/core/pipeline/features/pipeline_run.feature

@@ -0,0 +1,76 @@
+Feature: Pipeline running
+
+    Scenario Outline: Running a correct Pipeline
+        Given a pipeline <kind>
+        When I run the Pipeline
+        Then components are called with the expected inputs
+        And it should return the expected result
+
+        Examples:
+        | kind |
+        | that has no components |
+        | that is linear |
+        | that is really complex with lots of components, forks, and loops |
+        | that has a single component with a default input |
+        | that has two loops of identical lengths |
+        | that has two loops of different lengths |
+        | that has a single loop with two conditional branches |
+        | that has a component with dynamic inputs defined in init |
+        | that has two branches that don't merge |
+        | that has three branches that don't merge |
+        | that has two branches that merge |
+        | that has different combinations of branches that merge and do not merge |
+        | that has two branches, one of which loops back |
+        | that has a component with mutable input |
+        | that has a component with mutable output sent to multiple inputs |
+        | that has a greedy and variadic component after a component with default input |
+        | that has a component with only default inputs |
+        | that has a component with only default inputs as first to run and receives inputs from a loop |
+        | that has multiple branches that merge into a component with a single variadic input |
+        | that has multiple branches of different lengths that merge into a component with a single variadic input |
+        | that is linear and returns intermediate outputs |
+        | that has a loop and returns intermediate outputs from it |
+        | that is linear and returns intermediate outputs from multiple sockets |
+        | that has a component with default inputs that doesn't receive anything from its sender |
+        | that has a component with default inputs that doesn't receive anything from its sender but receives input from user |
+        | that has a loop and a component with default inputs that doesn't receive anything from its sender but receives input from user |
+        | that has multiple components with only default inputs and are added in a different order from the order of execution |
+        | that is linear with conditional branching and multiple joins |
+        | that is a simple agent |
+        | that has a variadic component that receives partial inputs |
+        | that has a variadic component that receives partial inputs in a different order |
+        | that has an answer joiner variadic component |
+        | that is linear and a component in the middle receives optional input from other components and input from the user |
+        | that has a loop in the middle |
+        | that has variadic component that receives a conditional input |
+        | that has a string variadic component |
+        | that is an agent that can use RAG |
+        | that has a feedback loop |
+        | created in a non-standard order that has a loop |
+        | that has an agent with a feedback cycle |
+        | that passes outputs that are consumed in cycle to outside the cycle |
+        | with a component that has dynamic default inputs |
+        | with a component that has variadic dynamic default inputs |
+        | that is a file conversion pipeline with two joiners |
+        | that is a file conversion pipeline with three joiners |
+        | that is a file conversion pipeline with three joiners and a loop |
+        | that has components returning dataframes |
+        | where a single component connects multiple sockets to the same receiver socket |
+        | where a component in a cycle provides inputs for a component outside the cycle in one iteration and no input in another iteration |
+        | that is blocked because not enough component inputs |
+        | that is a file conversion pipeline with three auto joiners |
+        | that has an auto joiner that takes in user inputs |
+        | that performs automatic conversion between list of ChatMessage and str |
+        | that performs automatic conversion wrapping ChatMessage for a Union receiver |
+
+    Scenario Outline: Running a bad Pipeline
+        Given a pipeline <kind>
+        When I run the Pipeline
+        Then it must have raised <exception>
+
+        Examples:
+        | kind | exception |
+        | that has an infinite loop | PipelineMaxComponentRuns |
+        | that has a component that doesn't return a dictionary | PipelineRuntimeError |
+        | that has a cycle that would get it stuck | PipelineComponentsBlockedError |
+        | that fails automatic conversion between list of ChatMessage and str | PipelineRuntimeError |