chore: import upstream snapshot with attribution
CodeQL / Analyze (python) (push) Has been cancelled
Update Platform Components Table / update (push) Has been cancelled
Docker image release / Build base image (push) Has been cancelled
Sync docs with Docusaurus / sync (push) Has been cancelled
Tests / Check if changed (push) Has been cancelled
Tests / format (push) Has been cancelled
Tests / check-imports (push) Has been cancelled
Tests / Unit / macos-latest (push) Has been cancelled
Tests / Unit / ubuntu-latest (push) Has been cancelled
Tests / Unit / windows-latest (push) Has been cancelled
Tests / mypy (push) Has been cancelled
Tests / Integration / ubuntu-latest (push) Has been cancelled
Tests / Integration / macos-latest (push) Has been cancelled
Tests / Integration / windows-latest (push) Has been cancelled
Tests / notify-slack-on-failure (push) Has been cancelled
Tests / Mark tests as completed (push) Has been cancelled
CodeQL / Analyze (python) (push) Has been cancelled
Update Platform Components Table / update (push) Has been cancelled
Docker image release / Build base image (push) Has been cancelled
Sync docs with Docusaurus / sync (push) Has been cancelled
Tests / Check if changed (push) Has been cancelled
Tests / format (push) Has been cancelled
Tests / check-imports (push) Has been cancelled
Tests / Unit / macos-latest (push) Has been cancelled
Tests / Unit / ubuntu-latest (push) Has been cancelled
Tests / Unit / windows-latest (push) Has been cancelled
Tests / mypy (push) Has been cancelled
Tests / Integration / ubuntu-latest (push) Has been cancelled
Tests / Integration / macos-latest (push) Has been cancelled
Tests / Integration / windows-latest (push) Has been cancelled
Tests / notify-slack-on-failure (push) Has been cancelled
Tests / Mark tests as completed (push) Has been cancelled
This commit is contained in:
@@ -0,0 +1,52 @@
|
||||
# Fuzz targets
|
||||
|
||||
[Atheris](https://github.com/google/atheris) fuzz harnesses for Haystack's
|
||||
untrusted-input entry points. They are wired into CI via
|
||||
[ClusterFuzzLite](https://google.github.io/clusterfuzzlite/)
|
||||
(see [`.clusterfuzzlite/`](../../.clusterfuzzlite) and the
|
||||
`ClusterFuzzLite PR fuzzing` workflow).
|
||||
|
||||
| Harness | Target | Why |
|
||||
|---|---|---|
|
||||
| `fuzz_pipeline_loads.py` | `Pipeline.loads` | Deserializing a serialized pipeline (YAML) is a documented attack surface. |
|
||||
| `fuzz_document_from_dict.py` | `Document.from_dict` | Reconstructing a `Document` from an untrusted dict. |
|
||||
| `fuzz_filters.py` | `document_matches_filter` | Evaluating an untrusted filter expression. |
|
||||
|
||||
Each harness catches the exceptions that are a *normal* reaction to malformed
|
||||
input; anything else (a crash, unbounded recursion, a hang, or an unexpected
|
||||
exception type) is reported by Atheris as a finding. The "expected" exception
|
||||
lists can be tightened over time to surface more subtle bugs.
|
||||
|
||||
## Seed corpus
|
||||
|
||||
`fuzz_document_from_dict` and `fuzz_filters` parse the raw fuzzer input as JSON,
|
||||
so the input domain *is* JSON text. A small seed corpus of valid inputs lives in
|
||||
[`corpus/<harness>/`](corpus) to bootstrap coverage past the JSON parse — without
|
||||
it a short run spends most of its budget producing inputs that don't parse. The
|
||||
ClusterFuzzLite build (`.clusterfuzzlite/build.sh`) zips each `corpus/<harness>/`
|
||||
into the `<harness>_seed_corpus.zip` the runner expects. Add a new valid input by
|
||||
dropping a `.json` file into the matching directory.
|
||||
|
||||
## Run locally
|
||||
|
||||
```sh
|
||||
pip install atheris
|
||||
pip install -e .
|
||||
|
||||
# Fuzz for a bit (Ctrl-C to stop); -atheris_runs limits the number of inputs.
|
||||
# Pass the seed corpus dir so libFuzzer starts from valid inputs.
|
||||
python test/fuzz/fuzz_pipeline_loads.py -atheris_runs=100000
|
||||
python test/fuzz/fuzz_document_from_dict.py test/fuzz/corpus/fuzz_document_from_dict -atheris_runs=100000
|
||||
python test/fuzz/fuzz_filters.py test/fuzz/corpus/fuzz_filters -atheris_runs=100000
|
||||
```
|
||||
|
||||
A directory argument is used as a seed corpus (and grown with new coverage); a
|
||||
crashing input file reproduces a finding:
|
||||
|
||||
```sh
|
||||
python test/fuzz/fuzz_filters.py crash-<hash> # reproduce a crash
|
||||
```
|
||||
|
||||
> Note: Atheris builds a native extension and is not part of the dev
|
||||
> dependencies; install it on demand as shown above. `pytest` does not collect
|
||||
> these files (they are named `fuzz_*.py`, not `test_*.py`).
|
||||
@@ -0,0 +1 @@
|
||||
{"content": "x", "page": 2, "name": "fuzz"}
|
||||
@@ -0,0 +1 @@
|
||||
{"content": "hello world"}
|
||||
@@ -0,0 +1 @@
|
||||
{"id": "doc-1", "content": "the quick brown fox", "meta": {"page": 1, "name": "fuzz"}}
|
||||
@@ -0,0 +1 @@
|
||||
{"id": "doc-2", "content": "x", "score": 0.5, "embedding": [0.1, 0.2, 0.3]}
|
||||
@@ -0,0 +1 @@
|
||||
{"field": "meta.page", "operator": "==", "value": 1}
|
||||
@@ -0,0 +1 @@
|
||||
{"field": "content", "operator": "==", "value": "the quick brown fox"}
|
||||
@@ -0,0 +1 @@
|
||||
{"operator": "AND", "conditions": [{"field": "meta.page", "operator": ">", "value": 0}, {"field": "meta.name", "operator": "==", "value": "fuzz"}]}
|
||||
@@ -0,0 +1 @@
|
||||
{"operator": "NOT", "conditions": [{"field": "meta.name", "operator": "in", "value": ["a", "b"]}]}
|
||||
@@ -0,0 +1,43 @@
|
||||
# SPDX-FileCopyrightText: 2022-present deepset GmbH <info@deepset.ai>
|
||||
#
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
|
||||
"""Fuzz target for ``Document.from_dict`` — deserializing untrusted document dicts."""
|
||||
|
||||
import json
|
||||
import sys
|
||||
|
||||
import atheris
|
||||
|
||||
with atheris.instrument_imports():
|
||||
from haystack.dataclasses import Document
|
||||
|
||||
# Normal reactions to malformed input; anything else is a genuine finding.
|
||||
_EXPECTED = (ValueError, TypeError, KeyError)
|
||||
|
||||
|
||||
def TestOneInput(data: bytes) -> None:
|
||||
"""Parse fuzzer bytes as a JSON object and feed it to ``Document.from_dict``."""
|
||||
# ``json.loads`` accepts ``bytes`` directly, so the raw fuzzer input *is* the JSON
|
||||
# text. This keeps the input domain identical to the seed corpus (see corpus/) and
|
||||
# lets libFuzzer mutate JSON directly instead of through a FuzzedDataProvider.
|
||||
try:
|
||||
obj = json.loads(data)
|
||||
except (ValueError, RecursionError, UnicodeDecodeError):
|
||||
return
|
||||
if not isinstance(obj, dict):
|
||||
return
|
||||
try:
|
||||
Document.from_dict(obj)
|
||||
except _EXPECTED:
|
||||
pass
|
||||
|
||||
|
||||
def main() -> None:
|
||||
"""Set up and run the Atheris fuzzing loop."""
|
||||
atheris.Setup(sys.argv, TestOneInput)
|
||||
atheris.Fuzz()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,48 @@
|
||||
# SPDX-FileCopyrightText: 2022-present deepset GmbH <info@deepset.ai>
|
||||
#
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
|
||||
"""Fuzz target for ``document_matches_filter`` — evaluating untrusted filter expressions."""
|
||||
|
||||
import json
|
||||
import sys
|
||||
|
||||
import atheris
|
||||
|
||||
with atheris.instrument_imports():
|
||||
from haystack.dataclasses import Document
|
||||
from haystack.errors import FilterError
|
||||
from haystack.utils.filters import document_matches_filter
|
||||
|
||||
# A fixed document is enough; we are fuzzing the filter expression, not the document.
|
||||
_DOCUMENT = Document(content="the quick brown fox", meta={"page": 1, "name": "fuzz"})
|
||||
|
||||
# Normal reactions to malformed filters; anything else is a genuine finding.
|
||||
_EXPECTED = (FilterError, ValueError, TypeError, KeyError)
|
||||
|
||||
|
||||
def TestOneInput(data: bytes) -> None:
|
||||
"""Parse fuzzer bytes as a JSON filter dict and evaluate it against a document."""
|
||||
# ``json.loads`` accepts ``bytes`` directly, so the raw fuzzer input *is* the JSON
|
||||
# text. This keeps the input domain identical to the seed corpus (see corpus/) and
|
||||
# lets libFuzzer mutate JSON directly instead of through a FuzzedDataProvider.
|
||||
try:
|
||||
filters = json.loads(data)
|
||||
except (ValueError, RecursionError, UnicodeDecodeError):
|
||||
return
|
||||
if not isinstance(filters, dict):
|
||||
return
|
||||
try:
|
||||
document_matches_filter(filters, _DOCUMENT)
|
||||
except _EXPECTED:
|
||||
pass
|
||||
|
||||
|
||||
def main() -> None:
|
||||
"""Set up and run the Atheris fuzzing loop."""
|
||||
atheris.Setup(sys.argv, TestOneInput)
|
||||
atheris.Fuzz()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,46 @@
|
||||
# SPDX-FileCopyrightText: 2022-present deepset GmbH <info@deepset.ai>
|
||||
#
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
|
||||
"""Fuzz target for ``Pipeline.loads`` — deserializing untrusted serialized pipelines.
|
||||
|
||||
Loading a serialized pipeline is an explicit attack surface (see SECURITY.md), so this
|
||||
target feeds arbitrary fuzzer bytes through the YAML unmarshaller and ``from_dict``.
|
||||
"""
|
||||
|
||||
import sys
|
||||
|
||||
import atheris
|
||||
|
||||
with atheris.instrument_imports():
|
||||
from haystack import Pipeline
|
||||
from haystack.core.errors import DeserializationError, PipelineError
|
||||
|
||||
# Exceptions that are a normal reaction to malformed input. Anything else — a crash,
|
||||
# unbounded recursion, a hang, or an unexpected exception type — is a genuine finding.
|
||||
_EXPECTED = (DeserializationError, PipelineError, ValueError, TypeError, KeyError)
|
||||
|
||||
# Known finding, deferred to a separate fix: non-dict YAML documents (e.g. an empty
|
||||
# string, a bare scalar, or a list) unmarshal cleanly and then make ``from_dict`` raise
|
||||
# a raw ``AttributeError`` from ``data.get(...)`` instead of a ``DeserializationError``.
|
||||
# Tolerated here so the target builds and fuzzes; remove once ``from_dict`` validates
|
||||
# its input type and this collapses back into ``DeserializationError``.
|
||||
_KNOWN_BUGS = (AttributeError,)
|
||||
|
||||
|
||||
def TestOneInput(data: bytes) -> None:
|
||||
"""Feed one fuzzer-generated input to ``Pipeline.loads`` as a YAML document."""
|
||||
try:
|
||||
Pipeline.loads(data)
|
||||
except _EXPECTED + _KNOWN_BUGS:
|
||||
pass
|
||||
|
||||
|
||||
def main() -> None:
|
||||
"""Set up and run the Atheris fuzzing loop."""
|
||||
atheris.Setup(sys.argv, TestOneInput)
|
||||
atheris.Fuzz()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
Reference in New Issue
Block a user