neuml--txtai/docs/pipeline/data/tokenizer.md

Tokenizer

The Tokenizer pipeline splits text into tokens. This is primarily used for keyword / term indexing.

Note: Transformers-based models have their own tokenizers and this pipeline isn't designed for working with Transformers models.

Example

The following shows a simple example using this pipeline.

from txtai.pipeline import Tokenizer

# Create and run pipeline
tokenizer = Tokenizer()
tokenizer("text to tokenize")

# Whitespace tokenization
tokenizer = Tokenizer(whitespace=True)
tokenizer("text to tokenize")

# Tokenize using a regular expression
tokenizer = Tokenizer(regexp=r"\w{5,}")
tokenizer("text to tokenize")

# Tokenize into trigrams like pg_trgm
tokenizer = Tokenizer(ngrams={
  "ngrams": 3, "lpad": "  ", "rpad": " ", "unique": True
})
tokenize("text to tokenize")

# Tokenize into edge ngrams
tokenizer = Tokenizer(ngrams={"nmin": 2, "nmax": 5, "edge": True})
tokenizer("text to tokenize")

Configuration-driven example

Pipelines are run with Python or configuration. Pipelines can be instantiated in configuration using the lower case name of the pipeline. Configuration-driven pipelines are run with workflows or the API.

config.yml

# Create pipeline using lower case class name
tokenizer:

# Run pipeline with workflow
workflow:
  tokenizer:
    tasks:
      - action: tokenizer

Run with Workflows

from txtai import Application

# Create and run pipeline with workflow
app = Application("config.yml")
list(app.workflow("tokenizer", ["text to tokenize"]))

Run with API

CONFIG=config.yml uvicorn "txtai.api:app" &

curl \
  -X POST "http://localhost:8000/workflow" \
  -H "Content-Type: application/json" \
  -d '{"name":"tokenizer", "elements":["text"]}'

Methods

Python documentation for the pipeline.

::: txtai.pipeline.Tokenizer.init

::: txtai.pipeline.Tokenizer.call