chore: import upstream snapshot with attribution
build / build (macos-latest) (push) Has been cancelled
build / build (ubuntu-latest) (push) Has been cancelled
build / build (windows-latest) (push) Has been cancelled
minimal / deploy (push) Has been cancelled

This commit is contained in:
wehub-resource-sync
2026-07-13 13:38:00 +08:00
commit 3a7c47b2a6
623 changed files with 133790 additions and 0 deletions
+33
View File
@@ -0,0 +1,33 @@
# Console Task
![task](../../images/task.png#only-light)
![task](../../images/task-dark.png#only-dark)
The Console Task prints task inputs and outputs to standard output. This task is mainly used for debugging and can be added at any point in a workflow.
## Example
The following shows a simple example using this task as part of a workflow.
```python
from txtai.workflow import FileTask, Workflow
workflow = Workflow([ConsoleTask()])
workflow(["Input 1", "Input2"])
```
## Configuration-driven example
This task can also be created with workflow configuration.
```yaml
workflow:
tasks:
- task: console
```
## Methods
Python documentation for the task.
### ::: txtai.workflow.ConsoleTask.__init__
+34
View File
@@ -0,0 +1,34 @@
# Export Task
![task](../../images/task.png#only-light)
![task](../../images/task-dark.png#only-dark)
The Export Task exports task outputs to CSV or Excel.
## Example
The following shows a simple example using this task as part of a workflow.
```python
from txtai.workflow import FileTask, Workflow
workflow = Workflow([ExportTask()])
workflow(["Input 1", "Input2"])
```
## Configuration-driven example
This task can also be created with workflow configuration.
```yaml
workflow:
tasks:
- task: export
```
## Methods
Python documentation for the task.
### ::: txtai.workflow.ExportTask.__init__
### ::: txtai.workflow.ExportTask.register
+33
View File
@@ -0,0 +1,33 @@
# File Task
![task](../../images/task.png#only-light)
![task](../../images/task-dark.png#only-dark)
The File Task validates a file exists. It handles both file paths and local file urls. Note that this task _only_ works with local files.
## Example
The following shows a simple example using this task as part of a workflow.
```python
from txtai.workflow import FileTask, Workflow
workflow = Workflow([FileTask()])
workflow(["/path/to/file", "file:///path/to/file"])
```
## Configuration-driven example
This task can also be created with workflow configuration.
```yaml
workflow:
tasks:
- task: file
```
## Methods
Python documentation for the task.
### ::: txtai.workflow.FileTask.__init__
+33
View File
@@ -0,0 +1,33 @@
# Image Task
![task](../../images/task.png#only-light)
![task](../../images/task-dark.png#only-dark)
The Image Task reads file paths, check the file is an image and opens it as an Image object. Note that this task _only_ works with local files.
## Example
The following shows a simple example using this task as part of a workflow.
```python
from txtai.workflow import ImageTask, Workflow
workflow = Workflow([ImageTask()])
workflow(["image.jpg", "image.gif"])
```
## Configuration-driven example
This task can also be created with workflow configuration.
```yaml
workflow:
tasks:
- task: image
```
## Methods
Python documentation for the task.
### ::: txtai.workflow.ImageTask.__init__
+84
View File
@@ -0,0 +1,84 @@
# Tasks
![task](../../images/task.png#only-light)
![task](../../images/task-dark.png#only-dark)
Workflows execute tasks. Tasks are callable objects with a number of parameters to control the processing of data at a given step. While similar to pipelines, tasks encapsulate processing and don't perform signficant transformations on their own. Tasks perform logic to prepare content for the underlying action(s).
A simple task is shown below.
```python
Task(lambda x: [y * 2 for y in x])
```
The task above executes the function above for all input elements.
Tasks work well with pipelines, since pipelines are callable objects. The example below will summarize each input element.
```python
summary = Summary()
Task(summary)
```
Tasks can operate independently but work best with workflows, as workflows add large-scale stream processing.
```python
summary = Summary()
task = Task(summary)
task(["Very long text here"])
workflow = Workflow([task])
list(workflow(["Very long text here"]))
```
Tasks can also be created with configuration as part of a workflow.
```yaml
workflow:
tasks:
- action: summary
```
::: txtai.workflow.Task.__init__
## Multi-action task concurrency
The default processing mode is to run actions sequentially. Multiprocessing support is already built in at a number of levels. Any of the GPU models will maximize GPU utilization for example and even in CPU mode, concurrency is utilized. But there are still use cases for task action concurrency. For example, if the system has multiple GPUs, the task runs external sequential code, or the task has a large number of I/O tasks.
In addition to sequential processing, multi-action tasks can run either multithreaded or with multiple processes. The advantages of each approach are discussed below.
- *multithreading* - no overhead of creating separate processes or pickling data. But Python can only execute a single thread due the GIL, so this approach won't help with CPU bound actions. This method works well with I/O bound actions and GPU actions.
- *multiprocessing* - separate subprocesses are created and data is exchanged via pickling. This method can fully utilize all CPU cores since each process runs independently. This method works well with CPU bound actions.
More information on multiprocessing can be found in the [Python documentation](https://docs.python.org/3/library/multiprocessing.html).
## Multi-action task merges
Multi-action tasks will generate parallel outputs for the input data. The task output can be merged together in a couple different ways.
### ::: txtai.workflow.Task.hstack
### ::: txtai.workflow.Task.vstack
### ::: txtai.workflow.Task.concat
## Extract task output columns
With column-wise merging, each output row will be a tuple of output values for each task action. This can be fed as input to a downstream task and that task can have separate tasks work with each element.
A simple example:
```python
workflow = Workflow([Task(lambda x: [y * 3 for y in x], unpack=False, column=0)])
list(workflow([(2, 8)]))
```
For the example input tuple of (2, 2), the workflow will only select the first element (2) and run the task against that element.
```python
workflow = Workflow([Task([lambda x: [y * 3 for y in x],
lambda x: [y - 1 for y in x]],
unpack=False, column={0:0, 1:1})])
list(workflow([(2, 8)]))
```
The example above applies a separate action to each input column. This simple construct can help build extremely powerful workflow graphs!
+35
View File
@@ -0,0 +1,35 @@
# Retrieve Task
![task](../../images/task.png#only-light)
![task](../../images/task-dark.png#only-dark)
The Retrieve Task connects to a url and downloads the content locally. This task is helpful when working with actions that require data to be available locally.
## Example
The following shows a simple example using this task as part of a workflow.
```python
from txtai.workflow import RetrieveTask, Workflow
workflow = Workflow([RetrieveTask(directory="/tmp")])
workflow(["https://file.to.download", "/local/file/to/copy"])
```
## Configuration-driven example
This task can also be created with workflow configuration.
```yaml
workflow:
tasks:
- task: retrieve
directory: /tmp
```
## Methods
Python documentation for the task.
### ::: txtai.workflow.RetrieveTask.__init__
### ::: txtai.workflow.RetrieveTask.register
+35
View File
@@ -0,0 +1,35 @@
# Service Task
![task](../../images/task.png#only-light)
![task](../../images/task-dark.png#only-dark)
The Service Task extracts content from a http service.
## Example
The following shows a simple example using this task as part of a workflow.
```python
from txtai.workflow import ServiceTask, Workflow
workflow = Workflow([ServiceTask(url="https://service.url/action)])
workflow(["parameter"])
```
## Configuration-driven example
This task can also be created with workflow configuration.
```yaml
workflow:
tasks:
- task: service
url: https://service.url/action
```
## Methods
Python documentation for the task.
### ::: txtai.workflow.ServiceTask.__init__
### ::: txtai.workflow.ServiceTask.register
+33
View File
@@ -0,0 +1,33 @@
# Storage Task
![task](../../images/task.png#only-light)
![task](../../images/task-dark.png#only-dark)
The Storage Task expands a local directory or cloud storage bucket into a list of URLs to process.
## Example
The following shows a simple example using this task as part of a workflow.
```python
from txtai.workflow import StorageTask, Workflow
workflow = Workflow([StorageTask()])
workflow(["s3://path/to/bucket", "local://local/directory"])
```
## Configuration-driven example
This task can also be created with workflow configuration.
```yaml
workflow:
tasks:
- task: storage
```
## Methods
Python documentation for the task.
### ::: txtai.workflow.StorageTask.__init__
+35
View File
@@ -0,0 +1,35 @@
# Template Task
![task](../../images/task.png#only-light)
![task](../../images/task-dark.png#only-dark)
The Template Task generates text from a template and task inputs. Templates can be used to prepare data for a number of tasks including generating large
language model (LLM) prompts.
## Example
The following shows a simple example using this task as part of a workflow.
```python
from txtai.workflow import TemplateTask, Workflow
workflow = Workflow([TemplateTask(template="This is a {text} task")])
workflow([{"text": "template"}])
```
## Configuration-driven example
This task can also be created with workflow configuration.
```yaml
workflow:
tasks:
- task: template
template: This is a {text} task
```
## Methods
Python documentation for the task.
### ::: txtai.workflow.TemplateTask.__init__
+33
View File
@@ -0,0 +1,33 @@
# Url Task
![task](../../images/task.png#only-light)
![task](../../images/task-dark.png#only-dark)
The Url Task validates that inputs start with a url prefix.
## Example
The following shows a simple example using this task as part of a workflow.
```python
from txtai.workflow import UrlTask, Workflow
workflow = Workflow([UrlTask()])
workflow(["https://file.to.download", "file:////local/file/to/copy"])
```
## Configuration-driven example
This task can also be created with workflow configuration.
```yaml
workflow:
tasks:
- task: url
```
## Methods
Python documentation for the task.
### ::: txtai.workflow.UrlTask.__init__
+23
View File
@@ -0,0 +1,23 @@
# Workflow Task
![task](../../images/task.png#only-light)
![task](../../images/task-dark.png#only-dark)
The Workflow Task runs a workflow. Allows creating workflows of workflows.
## Example
The following shows a simple example using this task as part of a workflow.
```python
from txtai.workflow import WorkflowTask, Workflow
workflow = Workflow([WorkflowTask(otherworkflow)])
workflow(["input data"])
```
## Methods
Python documentation for the task.
### ::: txtai.workflow.WorkflowTask.__init__