chore: import upstream snapshot with attribution

2026-07-13 13:22:28 +08:00
commit c56bef871b
9296 changed files with 1854228 additions and 0 deletions
@@ -0,0 +1,117 @@
+---
+title: "DocumentLanguageClassifier"
+id: documentlanguageclassifier
+slug: "/documentlanguageclassifier"
+description: "Use this component to classify documents by language and add language information to metadata."
+---
+
+# DocumentLanguageClassifier
+
+Use this component to classify documents by language and add language information to metadata.
+
+<div className="key-value-table">
+
+|  |  |
+| --- | --- |
+| **Most common position in a pipeline** | Before [`MetadataRouter`](../routers/metadatarouter.mdx)                                                                    |
+| **Mandatory run variables**            | `documents`:  A list of documents                                                                                  |
+| **Output variables**                   | `documents`:  A list of documents                                                                                  |
+| **API reference**                      | [Classifiers](/reference/classifiers-api)                                                                                 |
+| **GitHub link**                        | https://github.com/deepset-ai/haystack/blob/main/haystack/components/classifiers/document_language_classifier.py |
+
+</div>
+
+## Overview
+
+`DocumentLanguageClassifier` classifies the language of documents and adds the detected language to their metadata. If a document's text does not match any of the languages specified at initialization, it is classified as "unmatched". By default, the classifier classifies for English (”en”) documents, with the rest being classified as “unmatched”.
+
+The set of supported languages can be specified in the init method with the `languages` variable, using ISO codes.
+
+To route your documents to various branches of the pipeline based on the language, use `MetadataRouter` component right after `DocumentLanguageClassifier`.
+
+For classifying and then routing plain text using the same logic, use the `TextLanguageRouter` component instead.
+
+## Usage
+
+Install the `langdetect`package to use the `DocumentLanguageClassifier`component:
+
+```shell shell
+pip install langdetect
+```
+
+### On its own
+
+Below, we are using the `DocumentLanguageClassifier` to classify English and German documents:
+
+```python
+from haystack.components.classifiers import DocumentLanguageClassifier
+from haystack import Document
+
+documents = [
+    Document(content="Mein Name ist Jean und ich wohne in Paris."),
+    Document(content="Mein Name ist Mark und ich wohne in Berlin."),
+    Document(content="Mein Name ist Giorgio und ich wohne in Rome."),
+    Document(content="My name is Pierre and I live in Paris"),
+    Document(content="My name is Paul and I live in Berlin."),
+    Document(content="My name is Alessia and I live in Rome."),
+]
+
+document_classifier = DocumentLanguageClassifier(languages=["en", "de"])
+document_classifier.run(documents=documents)
+```
+
+### In a pipeline
+
+Below, we are using the `DocumentLanguageClassifier` in an indexing pipeline that indexes English and German documents into two difference indexes in an `InMemoryDocumentStore`, using embedding models for each language.
+
+```python
+from haystack import Pipeline
+from haystack import Document
+from haystack.document_stores.in_memory import InMemoryDocumentStore
+from haystack.components.classifiers import DocumentLanguageClassifier
+from haystack.components.embedders import SentenceTransformersDocumentEmbedder
+from haystack.components.writers import DocumentWriter
+from haystack.components.routers import MetadataRouter
+
+document_store_en = InMemoryDocumentStore()
+document_store_de = InMemoryDocumentStore()
+
+document_classifier = DocumentLanguageClassifier(languages=["en", "de"])
+metadata_router = MetadataRouter(
+    rules={"en": {"language": {"$eq": "en"}}, "de": {"language": {"$eq": "de"}}},
+)
+english_embedder = SentenceTransformersDocumentEmbedder()
+german_embedder = SentenceTransformersDocumentEmbedder(
+    model="PM-AI/bi-encoder_msmarco_bert-base_german",
+)
+en_writer = DocumentWriter(document_store=document_store_en)
+de_writer = DocumentWriter(document_store=document_store_de)
+
+indexing_pipeline = Pipeline()
+indexing_pipeline.add_component(
+    instance=document_classifier,
+    name="document_classifier",
+)
+indexing_pipeline.add_component(instance=metadata_router, name="metadata_router")
+indexing_pipeline.add_component(instance=english_embedder, name="english_embedder")
+indexing_pipeline.add_component(instance=german_embedder, name="german_embedder")
+indexing_pipeline.add_component(instance=en_writer, name="en_writer")
+indexing_pipeline.add_component(instance=de_writer, name="de_writer")
+
+indexing_pipeline.connect("document_classifier.documents", "metadata_router.documents")
+indexing_pipeline.connect("metadata_router.en", "english_embedder.documents")
+indexing_pipeline.connect("metadata_router.de", "german_embedder.documents")
+indexing_pipeline.connect("english_embedder", "en_writer")
+indexing_pipeline.connect("german_embedder", "de_writer")
+
+indexing_pipeline.run(
+    {
+        "document_classifier": {
+            "documents": [
+                Document(content="This is an English sentence."),
+                Document(content="Dies ist ein deutscher Satz."),
+            ],
+        },
+    },
+)
+```
@@ -0,0 +1,107 @@
+---
+title: "TransformersZeroShotDocumentClassifier"
+id: transformerszeroshotdocumentclassifier
+slug: "/transformerszeroshotdocumentclassifier"
+description: "Classifies the documents based on the provided labels and adds them to their metadata."
+---
+
+# TransformersZeroShotDocumentClassifier
+
+Classifies the documents based on the provided labels and adds them to their metadata.
+
+<div className="key-value-table">
+
+|  |  |
+| --- | --- |
+| **Most common position in a pipeline** | Before a [MetadataRouter](../routers/metadatarouter.mdx) |
+| **Mandatory init variables** | `model`: The name or path of a Hugging Face model for zero shot document classification  <br /> <br />`labels`: The set of possible class labels to classify each document into, for example, [`positive`, `negative`]. The labels depend on the selected model. |
+| **Mandatory run variables** | `documents`: A list of documents to classify |
+| **Output variables** | `documents`: A list of processed documents with an added `classification` metadata field |
+| **API reference** | [Classifiers](/reference/classifiers-api) |
+| **GitHub link** | https://github.com/deepset-ai/haystack/blob/main/haystack/components/classifiers/zero_shot_document_classifier.py |
+
+</div>
+
+## Overview
+
+The `TransformersZeroShotDocumentClassifier` component performs zero-shot classification of documents based on the labels that you set and adds the predicted label to their metadata.
+
+The component uses a Hugging Face pipeline for zero-shot classification.
+To initialize the component, provide the model and the set of labels to be used for categorization.
+You can additionally configure the component to allow multiple labels to be true with the `multi_label` boolean set to True.
+
+Classification is run on the document's content field by default. If you want it to run on another field, set the`classification_field` to one of the document's metadata fields.
+
+The classification results are stored in the `classification` dictionary within each document's metadata. If `multi_label` is set to `True`, you will find the scores for each label under the `details` key within the `classification` dictionary.
+
+Available models for the task of zero-shot-classification are:
+    - `valhalla/distilbart-mnli-12-3`
+    - `cross-encoder/nli-distilroberta-base`
+    - `cross-encoder/nli-deberta-v3-xsmall`
+
+## Usage
+
+### On its own
+
+```python
+from haystack import Document
+from haystack.components.classifiers import TransformersZeroShotDocumentClassifier
+
+documents = [
+    Document(id="0", content="Cats don't get teeth cavities."),
+    Document(id="1", content="Cucumbers can be grown in water."),
+]
+
+document_classifier = TransformersZeroShotDocumentClassifier(
+    model="cross-encoder/nli-deberta-v3-xsmall",
+    labels=["animals", "food"],
+)
+
+document_classifier.warm_up()
+document_classifier.run(documents=documents)
+```
+
+### In a pipeline
+
+The following is a pipeline that classifies documents based on predefined classification labels
+retrieved from a search pipeline:
+
+```python
+from haystack import Document
+from haystack.components.retrievers.in_memory import InMemoryBM25Retriever
+from haystack.document_stores.in_memory import InMemoryDocumentStore
+from haystack.core.pipeline import Pipeline
+from haystack.components.classifiers import TransformersZeroShotDocumentClassifier
+
+documents = [
+    Document(id="0", content="Today was a nice day!"),
+    Document(id="1", content="Yesterday was a bad day!"),
+]
+
+document_store = InMemoryDocumentStore()
+retriever = InMemoryBM25Retriever(document_store=document_store)
+document_classifier = TransformersZeroShotDocumentClassifier(
+    model="cross-encoder/nli-deberta-v3-xsmall",
+    labels=["positive", "negative"],
+)
+
+document_store.write_documents(documents)
+
+pipeline = Pipeline()
+pipeline.add_component(instance=retriever, name="retriever")
+pipeline.add_component(instance=document_classifier, name="document_classifier")
+pipeline.connect("retriever", "document_classifier")
+
+queries = ["How was your day today?", "How was your day yesterday?"]
+expected_predictions = ["positive", "negative"]
+
+for idx, query in enumerate(queries):
+    result = pipeline.run({"retriever": {"query": query, "top_k": 1}})
+    assert result["document_classifier"]["documents"][0].to_dict()["id"] == str(idx)
+    assert (
+        result["document_classifier"]["documents"][0].to_dict()["classification"][
+            "label"
+        ]
+        == expected_predictions[idx]
+    )
+```