neuml--txtai/docs/workflow/task/storage.md

Storage Task

The Storage Task expands a local directory or cloud storage bucket into a list of URLs to process.

Example

The following shows a simple example using this task as part of a workflow.

from txtai.workflow import StorageTask, Workflow

workflow = Workflow([StorageTask()])
workflow(["s3://path/to/bucket", "local://local/directory"])

Configuration-driven example

This task can also be created with workflow configuration.

workflow:
  tasks:
    - task: storage

Methods

Python documentation for the task.

::: txtai.workflow.StorageTask.init