chore: import upstream snapshot with attribution
This commit is contained in:
wehub-resource-sync
@@ -0,0 +1,136 @@
|
||||
# ANN
|
||||
|
||||
Approximate Nearest Neighbor (ANN) index configuration for storing vector embeddings.
|
||||
|
||||
## backend
|
||||
```yaml
|
||||
backend: faiss|hnsw|annoy|ggml|numpy|torch|turbovec|pgvector|sqlite|custom
|
||||
```
|
||||
|
||||
Sets the ANN backend. Defaults to `faiss`. Additional backends are available via the [ann](../../../install/#ann) extras package. Set custom backends via setting this parameter to the fully resolvable class string.
|
||||
|
||||
Backend-specific settings are set with a corresponding configuration object having the same name as the backend (i.e. annoy, faiss, or hnsw). These are optional and set to defaults if omitted.
|
||||
|
||||
### faiss
|
||||
```yaml
|
||||
faiss:
|
||||
components: comma separated list of components - defaults to "IDMap,Flat" for small
|
||||
indices and "IVFx,Flat" for larger indexes where
|
||||
x = min(4 * sqrt(embeddings count), embeddings count / 39)
|
||||
automatically calculates number of IVF cells when omitted (supports "IVF,Flat")
|
||||
nprobe: search probe setting (int) - defaults to x/16 (as defined above)
|
||||
for larger indexes
|
||||
nflip: same as nprobe - only used with binary hash indexes
|
||||
quantize: store vectors with x-bit precision vs 32-bit (boolean|int)
|
||||
true sets 8-bit precision, false disables, int sets specified
|
||||
precision
|
||||
mmap: load as on-disk index (boolean) - trade query response time for a
|
||||
smaller RAM footprint, defaults to false
|
||||
sample: percent of data to use for model training (0.0 - 1.0)
|
||||
reduces indexing time for larger (>1M+ row) indexes, defaults to 1.0
|
||||
```
|
||||
|
||||
Faiss supports both floating point and binary indexes. Floating point indexes are the default. Binary indexes are used when indexing scalar-quantized datasets.
|
||||
|
||||
See the following Faiss documentation links for more information.
|
||||
|
||||
- [Guidelines for choosing an index](https://github.com/facebookresearch/faiss/wiki/Guidelines-to-choose-an-index)
|
||||
- [Index configuration summary](https://github.com/facebookresearch/faiss/wiki/Faiss-indexes)
|
||||
- [Index Factory](https://github.com/facebookresearch/faiss/wiki/The-index-factory)
|
||||
- [Binary Indexes](https://github.com/facebookresearch/faiss/wiki/Binary-indexes)
|
||||
- [Search Tuning](https://github.com/facebookresearch/faiss/wiki/Faster-search)
|
||||
|
||||
Note: For macOS users, an existing bug in an upstream package restricts the number of processing threads to 1. This limitation is managed internally to prevent system crashes.
|
||||
|
||||
### hnsw
|
||||
```yaml
|
||||
hnsw:
|
||||
efconstruction: ef_construction param for init_index (int) - defaults to 200
|
||||
m: M param for init_index (int) - defaults to 16
|
||||
randomseed: random-seed param for init_index (int) - defaults to 100
|
||||
efsearch: ef search param (int) - defaults to None and not set
|
||||
```
|
||||
|
||||
See [Hnswlib documentation](https://github.com/nmslib/hnswlib/blob/master/ALGO_PARAMS.md) for more information on these parameters.
|
||||
|
||||
### annoy
|
||||
```yaml
|
||||
annoy:
|
||||
ntrees: number of trees (int) - defaults to 10
|
||||
searchk: search_k search setting (int) - defaults to -1
|
||||
```
|
||||
|
||||
See [Annoy documentation](https://github.com/spotify/annoy#full-python-api) for more information on these parameters. Note that annoy indexes can not be modified after creation, upserts/deletes and other modifications are not supported.
|
||||
|
||||
### ggml
|
||||
```yaml
|
||||
ggml:
|
||||
gpu: enable GPU - defaults to True
|
||||
quantize: sets the tensor quantization - defaults to F32
|
||||
querysize: query buffer size - defaults to 64
|
||||
```
|
||||
|
||||
The [GGML](https://github.com/ggml-org/ggml) backend is a k-nearest neighbors backend. It stores tensors using GGML and [GGUF](https://huggingface.co/docs/hub/en/gguf). It supports GPU-enabled operations and supports quantization. GGML is the framework used by [llama.cpp](https://github.com/ggml-org/llama.cpp).
|
||||
|
||||
[See this](https://github.com/ggml-org/ggml/blob/master/include/ggml.h#L379) for a list of quantization types.
|
||||
|
||||
### numpy
|
||||
|
||||
The NumPy backend is a k-nearest neighbors backend. It's designed for simplicity and works well with smaller datasets that fit into memory.
|
||||
|
||||
```yaml
|
||||
numpy:
|
||||
safetensors: stores vectors using the safetensors format
|
||||
defaults to NumPy array storage
|
||||
```
|
||||
|
||||
### torch
|
||||
|
||||
The Torch backend is a k-nearest neighbors backend like NumPy. It supports GPU-enabled operations. It also has support for quantization which enables fitting larger arrays into GPU memory.
|
||||
|
||||
When quantization is enabled, vectors are _always_ stored in safetensors. _Note that macOS support for quantization is limited._
|
||||
|
||||
```yaml
|
||||
torch:
|
||||
safetensors: stores vectors using the safetensors format - defaults
|
||||
to NumPy array storage if quantization is disabled
|
||||
quantize:
|
||||
type: quantization type (fp4, nf4, int8)
|
||||
blocksize: quantization block size parameter
|
||||
```
|
||||
|
||||
### turbovec
|
||||
|
||||
The [turbovec](https://github.com/RyanCodrai/turbovec) backend is a k-nearest neighbors backend powered by the [TurboQuant algorithm](https://arxiv.org/abs/2504.19874).
|
||||
|
||||
```yaml
|
||||
turbovec:
|
||||
bitwidth: number of bits to store each vector dimension as. Supports 2, 3 or 4.
|
||||
|
||||
### pgvector
|
||||
```yaml
|
||||
pgvector:
|
||||
url: database url connection string, alternatively can be set via
|
||||
ANN_URL environment variable
|
||||
schema: database schema to store vectors - defaults to being
|
||||
determined by the database
|
||||
table: database table to store vectors - defaults to `vectors`
|
||||
precision: vector float precision (half or full) - defaults to `full`
|
||||
efconstruction: ef_construction param (int) - defaults to 200
|
||||
m: M param for init_index (int) - defaults to 16
|
||||
```
|
||||
|
||||
The pgvector backend stores embeddings in a Postgres database. See the [pgvector documentation](https://github.com/pgvector/pgvector-python?tab=readme-ov-file#sqlalchemy) for more information on these parameters. See the [SQLAlchemy](https://docs.sqlalchemy.org/en/20/core/engines.html#database-urls) documentation for more information on how to construct url connection strings.
|
||||
|
||||
### sqlite
|
||||
```yaml
|
||||
sqlite:
|
||||
quantize: store vectors with x-bit precision vs 32-bit (boolean|int)
|
||||
true sets 8-bit precision, false disables, int sets specified
|
||||
precision
|
||||
table: database table to store vectors - defaults to `vectors`
|
||||
```
|
||||
|
||||
The SQLite backend stores embeddings in a SQLite database using [sqlite-vec](https://github.com/asg017/sqlite-vec). This backend supports 1-bit and 8-bit quantization at the storage level.
|
||||
|
||||
See [this note](https://alexgarcia.xyz/sqlite-vec/python.html#macos-blocks-sqlite-extensions-by-default) on how to run this ANN on MacOS.
|
||||
@@ -0,0 +1,110 @@
|
||||
# Cloud
|
||||
|
||||
The following describes parameters used to sync indexes with cloud storage. Cloud object storage, the [Hugging Face Hub](https://huggingface.co/models) and custom providers are all supported.
|
||||
|
||||
Parameters are set via the [embeddings.load](../../methods/#txtai.embeddings.base.Embeddings.load) and [embeddings.save](../../methods/#txtai.embeddings.base.Embeddings.save) methods.
|
||||
|
||||
## provider
|
||||
```yaml
|
||||
provider: string
|
||||
```
|
||||
|
||||
Cloud provider. Can be one of the following:
|
||||
|
||||
- Cloud object storage. Set to one of these [providers](https://libcloud.readthedocs.io/en/stable/storage/supported_providers.html). Use the text shown in the `Provider Constant` column as lower case.
|
||||
|
||||
- Hugging Face Hub. Set to `huggingface-hub`.
|
||||
|
||||
- Custom providers. Set to the full class path of the custom provider.
|
||||
|
||||
## container
|
||||
```yaml
|
||||
container: string
|
||||
```
|
||||
|
||||
Container/bucket/directory/repository name. Embeddings will be stored in the container with the filename specified by the `path` configuration.
|
||||
|
||||
## Cloud object storage configuration
|
||||
|
||||
In addition to the above common configuration, the cloud object storage provider has the following additional configuration parameters. Note that some cloud providers do not need any of these parameters and can use implicit authentication with service accounts.
|
||||
|
||||
See the [libcloud documentation](https://libcloud.readthedocs.io/en/stable/apidocs/libcloud.common.html#module-libcloud.common.base) for more information on these parameters.
|
||||
|
||||
### key
|
||||
```yaml
|
||||
key: string
|
||||
```
|
||||
|
||||
Provider-specific access key. Can also be set via `ACCESS_KEY` environment variable. Ensure the configuration file is secured if added to the file. When using implicit authentication, set this to a value such as 'using-implicit-auth'.
|
||||
|
||||
### secret
|
||||
```yaml
|
||||
secret: string
|
||||
```
|
||||
|
||||
Provider-specific access secret. Can also be set via `ACCESS_SECRET` environment variable. Ensure the configuration file is secured if added to the file. When using implicit authentication, this option is not required.
|
||||
|
||||
### prefix
|
||||
```yaml
|
||||
prefix: string
|
||||
```
|
||||
|
||||
Optional object prefix. Object storage doesn't have the concept of a directory but a prefix is similar. For example, a prefix could be `base/dir`. This helps with organizing data in an object storage bucket.
|
||||
|
||||
More can be found at the following links.
|
||||
|
||||
- [Organizing objects using prefixes](https://docs.aws.amazon.com/AmazonS3/latest/userguide/using-prefixes.html)
|
||||
- [libcloud container method documentation](https://libcloud.readthedocs.io/en/stable/storage/api.html#libcloud.storage.base.StorageDriver.iterate_container_objects)
|
||||
|
||||
### host
|
||||
```yaml
|
||||
host: string
|
||||
```
|
||||
|
||||
Optional server host name. Set when using a local cloud storage server.
|
||||
|
||||
### port
|
||||
```yaml
|
||||
port: int
|
||||
```
|
||||
|
||||
Optional server port. Set when using a local cloud storage server.
|
||||
|
||||
### token
|
||||
```yaml
|
||||
token: string
|
||||
```
|
||||
|
||||
Optional temporary session token
|
||||
|
||||
### region
|
||||
```yaml
|
||||
region: string
|
||||
```
|
||||
|
||||
Optional parameter to specify the storage region, provider-specific.
|
||||
|
||||
## Hugging Face Hub configuration
|
||||
|
||||
The huggingface-hub provider supports the following additional configuration parameters. More on these parameters can be found in the [Hugging Face Hub's documentation](https://huggingface.co/docs/huggingface_hub/main/en/package_reference/overview).
|
||||
|
||||
### revision
|
||||
```yaml
|
||||
revision: string
|
||||
```
|
||||
|
||||
Optional Git revision id which can be a branch name, a tag, or a commit hash
|
||||
|
||||
### cache
|
||||
```yaml
|
||||
cache: string
|
||||
```
|
||||
|
||||
Path to the folder where cached files are stored
|
||||
|
||||
### token
|
||||
```yaml
|
||||
token: string|boolean
|
||||
```
|
||||
|
||||
Token to be used for the download. If set to True, the token will be read from the Hugging Face config folder.
|
||||
@@ -0,0 +1,83 @@
|
||||
# Database
|
||||
|
||||
Databases store metadata, text and binary content.
|
||||
|
||||
## content
|
||||
```yaml
|
||||
content: boolean|sqlite|duckdb|client|url|custom
|
||||
```
|
||||
|
||||
Enables content storage. When true, the default storage engine, `sqlite` will be used to save metadata.
|
||||
|
||||
Client-server connections are supported with either `client` or a full connection URL. When set to `client`, the CLIENT_URL environment variable must be set to the full connection URL. See the [SQLAlchemy](https://docs.sqlalchemy.org/en/20/core/engines.html#database-urls) documentation for more information on how to construct connection strings for client-server databases.
|
||||
|
||||
Add custom storage engines via setting this parameter to the fully resolvable class string.
|
||||
|
||||
Content storage specific settings are set with a corresponding configuration object having the same name as the content storage engine (i.e. duckdb or sqlite). These are optional and set to defaults if omitted.
|
||||
|
||||
### client
|
||||
```yaml
|
||||
schema: default database schema for the session - defaults to being
|
||||
determined by the database
|
||||
```
|
||||
|
||||
Additional settings for client-server databases. Also supported when the `content=url`.
|
||||
|
||||
### sqlite
|
||||
```yaml
|
||||
sqlite:
|
||||
wal: enable write-ahead logging - allows concurrent read/write operations,
|
||||
defaults to false
|
||||
```
|
||||
|
||||
Additional settings for SQLite.
|
||||
|
||||
## objects
|
||||
```yaml
|
||||
objects: boolean|image|pickle
|
||||
```
|
||||
|
||||
Enables object storage. Supports storing binary content. Requires content storage to also be enabled.
|
||||
|
||||
Object encoding options are:
|
||||
|
||||
- `standard`: Default encoder when boolean set. Encodes and decodes objects as byte arrays.
|
||||
- `image`: Image encoder. Encodes and decodes objects as image objects.
|
||||
- `pickle`: Pickle encoder. Encodes and decodes objects with the pickle module. Supports arbitrary objects.
|
||||
|
||||
## functions
|
||||
```yaml
|
||||
functions: list
|
||||
```
|
||||
|
||||
List of functions with user-defined SQL functions. Each list element must be one of the following:
|
||||
|
||||
- function
|
||||
- callable object
|
||||
- dict with fields for name, argcount, function and deterministic
|
||||
|
||||
[An example can be found here](../../query#custom-sql-functions).
|
||||
|
||||
## expressions
|
||||
```yaml
|
||||
expressions: list
|
||||
```
|
||||
|
||||
List of expression shortcuts. Each list element must be a dict with the following fields.
|
||||
|
||||
- `name`: name of the expression
|
||||
- `expression`: SQL expression, defaults to `name` when empty
|
||||
- `index`: if this expression should have a database index, defaults to False when not provided
|
||||
|
||||
The expression can be a json data column, sql function or anything that can be run as a SQL snippet.
|
||||
|
||||
## query
|
||||
```yaml
|
||||
query:
|
||||
path: sets the path for the query model - this can be any model on the
|
||||
Hugging Face Model Hub or a local file path.
|
||||
prefix: text prefix to prepend to all inputs
|
||||
maxlength: maximum generated sequence length
|
||||
```
|
||||
|
||||
Query translation model. Translates natural language queries to txtai compatible SQL statements.
|
||||
@@ -0,0 +1,83 @@
|
||||
# General
|
||||
|
||||
General configuration options.
|
||||
|
||||
## keyword
|
||||
```yaml
|
||||
keyword: boolean|string
|
||||
```
|
||||
|
||||
Enables sparse keyword indexing for this embeddings.
|
||||
|
||||
When set to a boolean, this parameter creates a BM25 index for full text search. When set to a string, it expects a [keyword method](../scoring#method).
|
||||
|
||||
It also implicitly disables the [defaults](#defaults) setting for vector search.
|
||||
|
||||
## sparse
|
||||
```yaml
|
||||
sparse: boolean|path
|
||||
```
|
||||
|
||||
Enables sparse vector indexing for this embeddings.
|
||||
|
||||
When set to `True`, this parameter creates a sparse vector index using the [default sparse index model](https://huggingface.co/prithivida/Splade_PP_en_v2). When set to a string, it expects a local or Hugging Face model path.
|
||||
|
||||
It also implicitly disables the [defaults](#defaults) setting for vector search.
|
||||
|
||||
## dense
|
||||
```yaml
|
||||
dense: boolean|string
|
||||
```
|
||||
|
||||
Alias for the [vector model path](../vectors/#path). When set to `True`, the [default transformers vector model](https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2) is used.
|
||||
|
||||
## hybrid
|
||||
```yaml
|
||||
hybrid: boolean
|
||||
```
|
||||
|
||||
Enables hybrid (sparse + dense) indexing for this embeddings.
|
||||
|
||||
When enabled, this parameter creates a BM25 index for full text search. It has no effect on the [defaults](#defaults) or [path](../vectors/#path) settings.
|
||||
|
||||
## defaults
|
||||
```yaml
|
||||
defaults: boolean
|
||||
```
|
||||
|
||||
Uses default vector model path when enabled (default setting is True) and `path` is not provided. See [this link](../) for an example.
|
||||
|
||||
## indexes
|
||||
```yaml
|
||||
indexes: dict
|
||||
```
|
||||
|
||||
Key value pairs defining subindexes for this embeddings. Each key is the index name and the value is the full configuration. This configuration can use any of the available configurations in a standard embeddings instance.
|
||||
|
||||
## autoid
|
||||
```yaml
|
||||
format: int|uuid function
|
||||
```
|
||||
|
||||
Sets the auto id generation method. When this is not set, an autogenerated numeric sequence is used. This also supports [UUID generation functions](https://docs.python.org/3/library/uuid.html#uuid.uuid1). For example, setting this value to `uuid4` will generate random UUIDs. Setting this to `uuid5` will generate deterministic UUIDs for each input data row.
|
||||
|
||||
## columns
|
||||
```yaml
|
||||
columns:
|
||||
text: name of the text column
|
||||
object: name of the object column
|
||||
store: limit json data fields to this list of columns
|
||||
```
|
||||
|
||||
Sets the `text` and `object` column names. Defaults to `text` and `object` if not provided.
|
||||
|
||||
`store` sets a list of columns to store in the JSON data field. When this isn't provided, all columns are stored (default). When `store` is set to `None`, no JSON columns are stored. This is useful is a field is only needed at indexing time but not search time.
|
||||
|
||||
## format
|
||||
```yaml
|
||||
format: json|pickle
|
||||
```
|
||||
|
||||
Sets the configuration storage format. Defaults to `json`.
|
||||
|
||||
Note that `pickle` configuration is deprecated and will raise an error with the default settings. Its only kept for reading legacy indexes and requires setting the `ALLOW_PICKLE` environment variable. Only enable for local and/or trusted sources.
|
||||
@@ -0,0 +1,80 @@
|
||||
# Graph
|
||||
|
||||
Enable graph storage via the `graph` parameter. This component requires the [graph](../../../install/#graph) extras package.
|
||||
|
||||
When enabled, a graph network is built using the embeddings index. Graph nodes are synced with each embeddings index operation (index/upsert/delete). Graph edges are created using the embeddings index upon completion of each index/upsert/delete embeddings index call.
|
||||
|
||||
## backend
|
||||
```yaml
|
||||
backend: networkx|rdbms|custom
|
||||
```
|
||||
|
||||
Sets the graph backend. Defaults to `networkx`.
|
||||
|
||||
Add custom graph storage engines via setting this parameter to the fully resolvable class string.
|
||||
|
||||
The `rdbms` backend has the following additional settings.
|
||||
|
||||
### rdbms
|
||||
```yaml
|
||||
url: database url connection string, alternatively can be set via the
|
||||
GRAPH_URL environment variable
|
||||
schema: database schema to store graph - defaults to being
|
||||
determined by the database
|
||||
nodes: table to store node data, defaults to `nodes`
|
||||
edges: table to store edge data, defaults to `edges`
|
||||
```
|
||||
|
||||
## batchsize
|
||||
```yaml
|
||||
batchsize: int
|
||||
```
|
||||
|
||||
Batch query size, used to query embeddings index - defaults to 256.
|
||||
|
||||
## limit
|
||||
```yaml
|
||||
limit: int
|
||||
```
|
||||
|
||||
Maximum number of results to return per embeddings query - defaults to 15.
|
||||
|
||||
## minscore
|
||||
```yaml
|
||||
minscore: float
|
||||
```
|
||||
|
||||
Minimum score required to consider embeddings query matches - defaults to 0.1.
|
||||
|
||||
## approximate
|
||||
```yaml
|
||||
approximate: boolean
|
||||
```
|
||||
|
||||
When true, queries only run for nodes without edges - defaults to true.
|
||||
|
||||
## topics
|
||||
```yaml
|
||||
topics:
|
||||
algorithm: community detection algorithm (string), options are
|
||||
louvain (default), greedy, lpa
|
||||
level: controls number of topics (string), options are best (default) or first
|
||||
resolution: controls number of topics (int), larger values create more
|
||||
topics (int), defaults to 100
|
||||
labels: scoring index method used to build topic labels (string)
|
||||
options are bm25 (default), tfidf, sif
|
||||
terms: number of frequent terms to use for topic labels (int), defaults to 4
|
||||
stopwords: optional list of stop words to exclude from topic labels
|
||||
categories: optional list of categories used to group topics, allows
|
||||
granular topics with broad categories grouping topics
|
||||
```
|
||||
|
||||
Enables topic modeling. Defaults are tuned so that in most cases these values don't need to be changed (except for categories). These parameters are available for advanced use cases where one wants full control over the community detection process.
|
||||
|
||||
## copyattributes
|
||||
```yaml
|
||||
copyattributes: boolean|list
|
||||
```
|
||||
|
||||
Copy these attributes from input dictionaries in the `insert` method. If this is set to `True`, all attributes are copied. Otherwise, only the
|
||||
attributes specified in this list are copied to the graph as attributes.
|
||||
@@ -0,0 +1,51 @@
|
||||
# Configuration
|
||||
|
||||
The following describes available embeddings configuration. These parameters are set in the [Embeddings constructor](../methods#txtai.embeddings.base.Embeddings.__init__) via either the `config` parameter or as keyword arguments.
|
||||
|
||||
Configuration is designed to be optional and set only when needed. Out of the box, sensible defaults are picked to get up and running fast. For example:
|
||||
|
||||
```python
|
||||
from txtai import Embeddings
|
||||
|
||||
embeddings = Embeddings()
|
||||
```
|
||||
|
||||
Creates a new embeddings instance, using [all-MiniLM-L6-v2](https://hf.co/sentence-transformers/all-MiniLM-L6-v2) as the vector model, [Faiss](https://faiss.ai/) as the ANN index backend and content disabled.
|
||||
|
||||
```python
|
||||
from txtai import Embeddings
|
||||
|
||||
embeddings = Embeddings(content=True)
|
||||
```
|
||||
|
||||
Is the same as above except it adds in [SQLite](https://www.sqlite.org/index.html) for content storage.
|
||||
|
||||
The following sections link to all the available configuration options.
|
||||
|
||||
## [ANN](./ann)
|
||||
|
||||
The default vector index backend is Faiss.
|
||||
|
||||
## [Cloud](./cloud)
|
||||
|
||||
Embeddings databases can optionally be synced with cloud storage.
|
||||
|
||||
## [Database](./database)
|
||||
|
||||
Content storage is disabled by default. When enabled, SQLite is the default storage engine.
|
||||
|
||||
## [General](./general)
|
||||
|
||||
General configuration that doesn't fit elsewhere.
|
||||
|
||||
## [Graph](./graph)
|
||||
|
||||
An accomplying graph index can be created with an embeddings database. This enables topic modeling, path traversal and more. [NetworkX](https://github.com/networkx/networkx) is the default graph index.
|
||||
|
||||
## [Scoring](./scoring)
|
||||
|
||||
Sparse keyword indexing and word vectors term weighting.
|
||||
|
||||
## [Vectors](./vectors)
|
||||
|
||||
Vector search is enabled by converting text and other binary data into embeddings vectors. These vectors are then stored in an ANN index. The vector model is optional and a default model is used when not provided.
|
||||
@@ -0,0 +1,109 @@
|
||||
# Scoring
|
||||
|
||||
Enable scoring support via the `scoring` parameter.
|
||||
|
||||
This scoring instance can serve two purposes, depending on the settings.
|
||||
|
||||
One use case is building sparse/keyword indexes. This occurs when the `terms` parameter is set to `True`.
|
||||
|
||||
The other use case is with word vector term weighting. This feature has been available since the initial version but isn't quite as common anymore.
|
||||
|
||||
The following covers the available options.
|
||||
|
||||
## method
|
||||
```yaml
|
||||
method: bm25|tfidf|sif|pgtext|sparse|custom
|
||||
```
|
||||
|
||||
Sets the scoring method. Add custom scoring via setting this parameter to the fully resolvable class string.
|
||||
|
||||
### pgtext
|
||||
```yaml
|
||||
schema: database schema to store keyword index - defaults to being
|
||||
determined by the database
|
||||
```
|
||||
|
||||
Additional settings for Postgres full-text keyword indexes.
|
||||
|
||||
### sparse
|
||||
```yaml
|
||||
path: sparse vector model path
|
||||
vectormethod: vector embeddings method
|
||||
vectornormalize: enable vector embeddings normalization (boolean)
|
||||
gpu: boolean|int|string|device
|
||||
normalize: enable score normalization (boolean|float|string|dict)
|
||||
batch: Sets the transform batch size
|
||||
encodebatch: Sets the encode batch size
|
||||
vectors: additional model init args
|
||||
encodeargs: additional encode() args
|
||||
backend: ivfsparse|pgsparse
|
||||
```
|
||||
|
||||
Sparse vector scoring options. The sparse scoring instance combines a sparse vector model with a sparse approximate nearest neighbor index (ANN). This method supports both vector normalization and score normalization.
|
||||
|
||||
Vector normalization normalizes all vectors to have a magnitude of 1. By extension, all generated scores will be 0 to 1.
|
||||
|
||||
Score normalization scales the output between 0 and 1. This setting supports:
|
||||
|
||||
- `True` for default scale normalization
|
||||
- `float` normalize using this as the scale factor
|
||||
- `"bayes"` for Bayesian normalization using dynamic candidate score statistics
|
||||
- `{method: "bayes", alpha: 1.0, beta: null}` for Bayesian normalization with optional custom parameters
|
||||
|
||||
#### ivfsparse
|
||||
```yaml
|
||||
ivfsparse:
|
||||
sample: percent of data to use for model training (0.0 - 1.0)
|
||||
nfeatures: top n features to use for model training (int)
|
||||
nlist: desired number of clusters (int)
|
||||
nprobe: search probe setting (int)
|
||||
minpoints: minimum number of points for a cluster (int)
|
||||
```
|
||||
|
||||
Inverted file (IVF) index with flat vector file storage and sparse array support.
|
||||
|
||||
#### pgsparse
|
||||
|
||||
Sparse ANN backed by Postgres. Supports same options as the [pgvector](../ann/#pgvector) ANN.
|
||||
|
||||
## terms
|
||||
```yaml
|
||||
terms: boolean|dict
|
||||
```
|
||||
|
||||
Enables term frequency sparse arrays for a scoring instance. This is the backend for sparse keyword indexes.
|
||||
|
||||
Supports a `dict` with the parameters `cachelimit` and `cutoff`.
|
||||
|
||||
`cachelimit` is the maximum amount of resident memory in bytes to use during indexing before flushing to disk. This parameter is an `int`.
|
||||
|
||||
`cutoff` is used during search to determine what constitutes a common term. This parameter is a `float`, i.e. 0.1 for a cutoff of 10%.
|
||||
|
||||
When `terms` is set to `True`, default parameters are used for the `cachelimit` and `cutoff`. Normally, these defaults are sufficient.
|
||||
|
||||
## normalize
|
||||
```yaml
|
||||
normalize: boolean|str|dict
|
||||
```
|
||||
|
||||
Enables normalized scoring (ranging from 0 to 1). This setting supports:
|
||||
|
||||
- `True` for standard score normalization
|
||||
- `"bayes"` | `"bb25"` for Bayesian normalization using dynamic candidate score statistics
|
||||
- `{method: "bayes", alpha: 1.0, beta: null}` for Bayesian normalization with optional custom parameters
|
||||
|
||||
When standard normalization is enabled, statistics from the index are used to calculate normalized scores.
|
||||
When Bayesian/BB25 normalization is enabled, it uses positive-score candidates, dynamic `beta=median(scores)`, adaptive
|
||||
`alpha_eff=alpha/std(scores)` and a sigmoid transform (likelihood-only variant with flat prior) to map scores to `[0, 1]`.
|
||||
|
||||
Bayesian normalization references:
|
||||
|
||||
- [https://github.com/instructkr/bb25](https://github.com/instructkr/bb25)
|
||||
- [https://github.com/cognica-io/bayesian-bm25](https://github.com/cognica-io/bayesian-bm25)
|
||||
|
||||
## tokenizer
|
||||
```yaml
|
||||
tokenizer: dict
|
||||
```
|
||||
|
||||
Set tokenization rules. Passes these arguments to the underlying [Tokenization pipeline](../../../pipeline/data/tokenizer#txtai.pipeline.Tokenizer.__init__).
|
||||
@@ -0,0 +1,161 @@
|
||||
# Vectors
|
||||
|
||||
The following covers available vector model configuration options.
|
||||
|
||||
## path
|
||||
```yaml
|
||||
path: string
|
||||
```
|
||||
|
||||
Sets the path for a vectors model. When using a transformers/sentence-transformers model, this can be any model on the
|
||||
[Hugging Face Hub](https://huggingface.co/models) or a local file path. Otherwise, it must be a local file path to a word embeddings model.
|
||||
|
||||
## method
|
||||
```yaml
|
||||
method: transformers|sentence-transformers|llama.cpp|litellm|model2vec|external|
|
||||
words
|
||||
```
|
||||
|
||||
Embeddings method to use. If the method is not provided, it is inferred using the `path`.
|
||||
|
||||
`sentence-transformers`, `llama.cpp`, `litellm`, `model2vec` and `words` require the [vectors](../../../install/#vectors) extras package to be installed.
|
||||
|
||||
### transformers
|
||||
|
||||
Builds embeddings using a transformers model. While this can be any transformers model, it works best with
|
||||
[models trained](https://huggingface.co/models?pipeline_tag=sentence-similarity) to build embeddings.
|
||||
|
||||
`mean`, `cls`, `last` and `late` pooling are supported and automatically inferred from the model. The pooling method can be overwritten by changing the method
|
||||
from `transformers` to `meanpooling`, `clspooling`, `lastpooling` or `latepooling` respectively.
|
||||
|
||||
Setting `maxlength` to `True` enables truncating inputs to the `max_seq_length`. Setting `maxlength` to an integer will truncate inputs to that value. When omitted (default), the `maxlength` will be set to either the model or tokenizer maxlength.
|
||||
|
||||
Supports loading ONNX models directly. [See this section for more](../../../examples/#model-training).
|
||||
|
||||
### sentence-transformers
|
||||
|
||||
Same as transformers but loads models with the [sentence-transformers](https://github.com/UKPLab/sentence-transformers) library.
|
||||
|
||||
### llama.cpp
|
||||
|
||||
Builds embeddings using a [llama.cpp](https://github.com/abetlen/llama-cpp-python) model. Supports both local and remote GGUF paths on the HF Hub.
|
||||
|
||||
### litellm
|
||||
|
||||
Builds embeddings using a LiteLLM model. See the [LiteLLM documentation](https://litellm.vercel.app/docs/providers) for the options available with LiteLLM models.
|
||||
|
||||
### model2vec
|
||||
|
||||
Builds embeddings using a [Model2Vec](https://github.com/MinishLab/model2vec) model. Model2Vec is a knowledge-distilled version of a transformers model with static vectors.
|
||||
|
||||
### words
|
||||
|
||||
Builds embeddings using a word embeddings model and static vectors. While Transformers models are preferred in most cases, this method can be useful for low resource and historical languages where there isn't much linguistic data available.
|
||||
|
||||
#### pca
|
||||
```yaml
|
||||
pca: int
|
||||
```
|
||||
|
||||
Removes _n_ principal components from generated embeddings. When enabled, a TruncatedSVD model is built to help with dimensionality reduction. After pooling of vectors creates a single embedding, this method is applied.
|
||||
|
||||
### external
|
||||
|
||||
Embeddings are created via an external model or API. Requires setting the [transform](#transform) parameter to a function that translates data into embeddings.
|
||||
|
||||
#### transform
|
||||
```yaml
|
||||
transform: function
|
||||
```
|
||||
|
||||
When method is `external`, this function transforms input content into embeddings. The input to this function is a list of data. This method must return either a numpy array or list of numpy arrays.
|
||||
|
||||
## gpu
|
||||
```yaml
|
||||
gpu: boolean|int|string|device
|
||||
```
|
||||
|
||||
Set the target device. Supports true/false, device id, device string and torch device instance. This is automatically derived if omitted.
|
||||
|
||||
The `sentence-transformers` method supports encoding with multiple GPUs. This can be enabled by setting the gpu parameter to `all`.
|
||||
|
||||
## batch
|
||||
```yaml
|
||||
batch: int
|
||||
```
|
||||
|
||||
Sets the transform batch size. This parameter controls how input streams are chunked and vectorized.
|
||||
|
||||
## encodebatch
|
||||
```yaml
|
||||
encodebatch: int
|
||||
```
|
||||
|
||||
Sets the encode batch size. This parameter controls the underlying vector model batch size. This often corresponds to a GPU batch size, which controls GPU memory usage.
|
||||
|
||||
## dimensionality
|
||||
```yaml
|
||||
dimensionality: int
|
||||
```
|
||||
|
||||
Enables truncation of vectors to this dimensionality. This is only useful for models trained to store more important information in earlier dimensions such as [Matryoshka Representation Learning (MRL)](https://huggingface.co/blog/matryoshka).
|
||||
|
||||
## quantize
|
||||
```yaml
|
||||
quantize: int|boolean
|
||||
```
|
||||
|
||||
Enables scalar vector quantization at the specified precision. Supports 1-bit through 8-bit quantization. Scalar quantization transforms continuous floating point values to discrete unsigned integers. The `faiss`, `pgvector`, `numpy` and `torch` ANN backends support storing these vectors.
|
||||
|
||||
This parameter supports booleans for backwards compatability. When set to true/false, this flag sets [faiss.quantize](../ann/#faiss).
|
||||
|
||||
In addition to vector-level quantization, some ANN backends have the ability to quantize vectors at the storage layer. See the [ANN](../ann) configuration options for more.
|
||||
|
||||
## instructions
|
||||
```yaml
|
||||
instructions:
|
||||
query: prefix for queries
|
||||
data: prefix for indexing
|
||||
```
|
||||
|
||||
Instruction-based models use prefixes to modify how embeddings are computed. This is especially useful with asymmetric search, which is when the query and indexed data are of vastly different lengths. In other words, short queries with long documents.
|
||||
|
||||
`txtai` automatically loads prompts stored in `config_sentence_transformers.json` except if this parameter is set. For some older models such as [E5-base](https://huggingface.co/intfloat/e5-base), instructions still need to be provided via this parameter.
|
||||
|
||||
## models
|
||||
```yaml
|
||||
models: dict
|
||||
```
|
||||
|
||||
Loads and stores vector models in this cache. This is primarily used with subindexes but can be set on any embeddings instance. This prevents the same model from being loaded multiple times when working with multiple embeddings instances.
|
||||
|
||||
## tokenize
|
||||
```yaml
|
||||
tokenize: boolean
|
||||
```
|
||||
|
||||
Enables string tokenization (defaults to false). This method applies tokenization rules that only work with English language text. It's not recommended for use with recent vector models.
|
||||
|
||||
## vectors
|
||||
```yaml
|
||||
vectors: dict
|
||||
```
|
||||
|
||||
Passes these additional parameters to the underlying vector model.
|
||||
|
||||
### muvera
|
||||
```yaml
|
||||
muvera:
|
||||
repetitions: defaults 20
|
||||
hashes: defaults to 5
|
||||
projection: defaults 16
|
||||
```
|
||||
|
||||
Settings to control the size of MUVERA fixed dimensional outputs. Default is 20 * 2^5 * 16 = 10,240 dimensions.
|
||||
|
||||
### trust_remote_code
|
||||
```yaml
|
||||
trust_remote_code: boolean
|
||||
```
|
||||
|
||||
Parameter for trusting the code from Hugging Face models with custom implementations.
|
||||
@@ -0,0 +1,59 @@
|
||||
# Index format
|
||||
|
||||
|
||||
|
||||
|
||||
This section documents the txtai index format. Each component is designed to ensure open access to the underlying data in a programmatic and platform independent way
|
||||
|
||||
If an underlying library has an index format, that is used. Otherwise, txtai persists content with [MessagePack](https://msgpack.org/index.html) serialization.
|
||||
|
||||
To learn more about how these components work together, read the [Index Guide](../indexing) and [Query Guide](../query).
|
||||
|
||||
## ANN
|
||||
|
||||
Approximate Nearest Neighbor (ANN) index configuration for storing vector embeddings.
|
||||
|
||||
| Component | Storage Format |
|
||||
| ------------------------------------------------------------- | ---------------------------------------------------------------------------- |
|
||||
| [Faiss](https://github.com/facebookresearch/faiss) | Local file format provided by library |
|
||||
| [Hnswlib](https://github.com/nmslib/hnswlib) | Local file format provided by library |
|
||||
| [Annoy](https://github.com/spotify/annoy) | Local file format provided by library |
|
||||
| [NumPy](https://github.com/numpy/numpy) | Local NumPy array files via np.save / np.load |
|
||||
| [Postgres via pgvector](https://github.com/pgvector/pgvector) | Vector tables in a Postgres database |
|
||||
|
||||
## Core
|
||||
|
||||
Core embeddings index files.
|
||||
|
||||
| Component | Storage Format |
|
||||
| ------------------------------------------------------------- | ---------------------------------------------------------------------------- |
|
||||
| [Configuration](https://www.json.org/) | Embeddings index configuration stored as JSON |
|
||||
| [Index Ids](https://msgpack.org/index.html) | Embeddings index ids serialized with MessagePack. Only enabled when when content storage (database) is disabled. |
|
||||
|
||||
## Database
|
||||
|
||||
Databases store metadata, text and binary content.
|
||||
|
||||
| Component | Storage Format |
|
||||
| ------------------------------------------------------------- | ---------------------------------------------------------------------------- |
|
||||
| [SQLite](https://www.sqlite.org/) | Local database files with SQLite |
|
||||
| [DuckDB](https://github.com/duckdb/duckdb) | Local database files with DuckDB |
|
||||
| [Postgres](https://www.postgresql.org/) | Postgres relational database via [SQLAlchemy](https://github.com/sqlalchemy/sqlalchemy). Supports additional databases via this library. |
|
||||
|
||||
## Graph
|
||||
|
||||
Graph nodes and edges for an embeddings index
|
||||
|
||||
| Component | Storage Format |
|
||||
| ------------------------------------------------------------- | ----------------------------------------------------------------------------- |
|
||||
| [NetworkX](https://github.com/networkx/networkx) | Nodes and edges exported to local file serialized with MessagePack |
|
||||
| [Postgres](https://github.com/aplbrain/grand) | Nodes and edges stored in a Postgres database. Supports additional databases. |
|
||||
|
||||
## Scoring
|
||||
|
||||
Sparse/keyword indexing
|
||||
|
||||
| Component | Storage Format |
|
||||
| ------------------------------------------------------------- | ----------------------------------------------------------------------------- |
|
||||
| [Local index](https://www.sqlite.org/) | Metadata serialized with MessagePack. Terms stored in SQLite. |
|
||||
| [Postgres](https://www.postgresql.org/docs/current/textsearch.html) | Text indexed with Postgres Full Text Search (FTS) |
|
||||
@@ -0,0 +1,122 @@
|
||||
# Embeddings
|
||||
|
||||
|
||||
|
||||
|
||||
Embeddings databases are the engine that delivers semantic search. Data is transformed into embeddings vectors where similar concepts will produce similar vectors. Indexes both large and small are built with these vectors. The indexes are used to find results that have the same meaning, not necessarily the same keywords.
|
||||
|
||||
The following code snippet shows how to build and search an embeddings index.
|
||||
|
||||
```python
|
||||
from txtai import Embeddings
|
||||
|
||||
# Create embeddings model, backed by sentence-transformers & transformers
|
||||
embeddings = Embeddings(path="sentence-transformers/nli-mpnet-base-v2")
|
||||
|
||||
data = [
|
||||
"US tops 5 million confirmed virus cases",
|
||||
"Canada's last fully intact ice shelf has suddenly collapsed, " +
|
||||
"forming a Manhattan-sized iceberg",
|
||||
"Beijing mobilises invasion craft along coast as Taiwan tensions escalate",
|
||||
"The National Park Service warns against sacrificing slower friends " +
|
||||
"in a bear attack",
|
||||
"Maine man wins $1M from $25 lottery ticket",
|
||||
"Make huge profits without work, earn up to $100,000 a day"
|
||||
]
|
||||
|
||||
# Index the list of text
|
||||
embeddings.index(data)
|
||||
|
||||
print(f"{'Query':20} Best Match")
|
||||
print("-" * 50)
|
||||
|
||||
# Run an embeddings search for each query
|
||||
for query in ("feel good story", "climate change", "public health story", "war",
|
||||
"wildlife", "asia", "lucky", "dishonest junk"):
|
||||
# Extract uid of first result
|
||||
# search result format: (uid, score)
|
||||
uid = embeddings.search(query, 1)[0][0]
|
||||
|
||||
# Print text
|
||||
print(f"{query:20} {data[uid]}")
|
||||
```
|
||||
|
||||
## Build
|
||||
|
||||
An embeddings instance is [configuration-driven](configuration) based on what is passed in the constructor. Vectors are stored with the option to also [store content](configuration/database#content). Content storage enables additional filtering and data retrieval options.
|
||||
|
||||
The example above sets a specific embeddings vector model via the [path](configuration/vectors/#path) parameter. An embeddings instance with no configuration can also be created.
|
||||
|
||||
```python
|
||||
embeddings = Embeddings()
|
||||
```
|
||||
|
||||
In this case, when loading and searching for data, the [default transformers vector model](https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2) is used to vectorize data. See the [model guide](../models) for current model recommentations.
|
||||
|
||||
## Index
|
||||
|
||||
After creating a new embeddings instance, the next step is adding data to it.
|
||||
|
||||
```python
|
||||
embeddings.index(rows)
|
||||
```
|
||||
|
||||
The index method takes an iterable and supports the following formats for each element.
|
||||
|
||||
- `(id, data, tags)` - default processing format
|
||||
|
||||
| Element | Description |
|
||||
| ----------- | ------------------------------------------------------------- |
|
||||
| id | unique record id |
|
||||
| data | input data to index, can be text, a dictionary or object |
|
||||
| tags | optional tags string, used to mark/label data as it's indexed |
|
||||
|
||||
- `(id, data)`
|
||||
|
||||
Same as above but without tags.
|
||||
|
||||
- `data`
|
||||
|
||||
Single element to index. In this case, unique id's will automatically be generated. Note that for generated id's, [upsert](methods/#txtai.embeddings.base.Embeddings.upsert) and [delete](methods/#txtai.embeddings.base.Embeddings.delete) calls require a separate search to get the target ids.
|
||||
|
||||
When the data field is a dictionary, text is passed via the `text` key, binary objects via the `object` key. Note that [content](configuration/database#content) must be enabled to store metadata and [objects](configuration/database#objects) to store binary object data. The `id` and `tags` keys will be extracted, if provided.
|
||||
|
||||
The input iterable can be a list or generator. [Generators](https://wiki.python.org/moin/Generators) help with indexing very large datasets as only portions of the data is in memory at any given time.
|
||||
|
||||
More information on indexing can be found in the [index guide](indexing).
|
||||
|
||||
## Search
|
||||
|
||||
Once data is indexed, it is ready for search.
|
||||
|
||||
```python
|
||||
embeddings.search(query, limit)
|
||||
```
|
||||
|
||||
The search method takes two parameters, the query and query limit. The results format is different based on whether [content](configuration/database#content) is stored or not.
|
||||
|
||||
- List of `(id, score)` when content is _not_ stored
|
||||
- List of `{**query columns}` when content is stored
|
||||
|
||||
Both natural language and SQL queries are supported. More information can be found in the [query guide](query).
|
||||
|
||||
## Resource management
|
||||
|
||||
Embeddings databases are context managers. The following blocks automatically [close](methods/#txtai.embeddings.base.Embeddings.close) and free resources upon completion.
|
||||
|
||||
```python
|
||||
# Create a new Embeddings database, index data and save
|
||||
with Embeddings() as embeddings:
|
||||
embeddings.index(rows)
|
||||
embeddings.save(path)
|
||||
|
||||
# Search a saved Embeddings database
|
||||
with Embeddings().load(path) as embeddings:
|
||||
embeddings.search(query)
|
||||
```
|
||||
|
||||
While calling `close` isn't always necessary (resources will be garbage collected), it's best to free shared resources like database connections as soon as they aren't needed.
|
||||
|
||||
## More examples
|
||||
|
||||
See [this link](../examples/#semantic-search) for a full list of embeddings examples.
|
||||
@@ -0,0 +1,136 @@
|
||||
# Index guide
|
||||
|
||||
|
||||
|
||||
|
||||
This section gives an in-depth overview on how to index data with txtai. We'll cover vectorization, indexing/updating/deleting data and the various components of an embeddings database.
|
||||
|
||||
## Vectorization
|
||||
|
||||
The most compute intensive step in building an index is vectorization. The [path](../configuration/vectors#path) parameter sets the path to the vector model. There is logic to automatically detect the vector model [method](../configuration/vectors#method) but it can also be set directly.
|
||||
|
||||
The [batch](../configuration/vectors#batch) and [encodebatch](../configuration/vectors#encodebatch) parameters control the vectorization process. Larger values for `batch` will pass larger batches to the vectorization method. Larger values for `encodebatch` will pass larger batches for each vector encode call. In the case of GPU vector models, larger values will consume more GPU memory.
|
||||
|
||||
Data is buffered to temporary storage during indexing as embeddings vectors can be quite large (for example 768 dimensions of float32 is 768 * 4 = 3072 bytes per vector). Once vectorization is complete, a mmapped array is created with all vectors for [Approximate Nearest Neighbor (ANN)](../configuration/vectors#backend) indexing.
|
||||
|
||||
The terms `ANN` and `dense vector index` are used interchangeably throughout txtai's documentation.
|
||||
|
||||
## Setting a backend
|
||||
|
||||
As mentioned above, computed vectors are stored in an ANN. There are various index [backends](../configuration/ann#backend) that can be configured. Faiss is the default backend.
|
||||
|
||||
## Content storage
|
||||
|
||||
Embeddings indexes can optionally [store content](../configuration/database#content). When this is enabled, the input content is saved in a database alongside the computed vectors. This enables filtering on additional fields and content retrieval.
|
||||
|
||||
The columns used for text, object and JSON data storage are set via [column configuration](../configuration/general#columns).
|
||||
|
||||
## Index vs Upsert
|
||||
|
||||
Data is loaded into an index with either an [index](../methods#txtai.embeddings.base.Embeddings.index) or [upsert](../methods#txtai.embeddings.base.Embeddings.upsert) call.
|
||||
|
||||
```python
|
||||
embeddings.index([(uid, text, None) for uid, text in enumerate(data)])
|
||||
embeddings.upsert([(uid, text, None) for uid, text in enumerate(data)])
|
||||
```
|
||||
|
||||
The `index` call will build a brand new index replacing an existing one. `upsert` will insert or update records. `upsert` ops do _not_ require a full index rebuild.
|
||||
|
||||
## Save
|
||||
|
||||
Indexes can be stored in a directory using the [save](../methods/#txtai.embeddings.base.Embeddings.save) method.
|
||||
|
||||
```python
|
||||
embeddings.save("/path/to/save")
|
||||
```
|
||||
|
||||
Compressed indexes are also supported.
|
||||
|
||||
```python
|
||||
embeddings.save("/path/to/save/index.tar.gz")
|
||||
```
|
||||
|
||||
In addition to saving indexes locally, they can also be persisted to [cloud storage](../configuration/cloud).
|
||||
|
||||
```python
|
||||
embeddings.save("/path/to/save/index.tar.gz", cloud={...})
|
||||
```
|
||||
|
||||
This is especially useful when running in a serverless context or otherwise running on temporary compute. Cloud storage is only supported with compressed indexes.
|
||||
|
||||
Embeddings indexes can be restored using the [load](../methods/#txtai.embeddings.base.Embeddings.load) method.
|
||||
|
||||
```python
|
||||
embeddings.load("/path/to/load")
|
||||
```
|
||||
|
||||
## Delete
|
||||
|
||||
Content can be removed from the index with the [delete](../methods#txtai.embeddings.base.Embeddings.delete) method. This method takes a list of ids to delete.
|
||||
|
||||
```python
|
||||
embeddings.delete(ids)
|
||||
```
|
||||
|
||||
## Reindex
|
||||
|
||||
When [content storage](../configuration/database#content) is enabled, [reindex](../methods#txtai.embeddings.base.Embeddings.reindex) can be called to rebuild the index with new settings. For example, the backend can be switched from faiss to hnsw or the vector model can be updated. This prevents having to go back to the original raw data.
|
||||
|
||||
```python
|
||||
embeddings.reindex(path="sentence-transformers/all-MiniLM-L6-v2", backend="hnsw")
|
||||
```
|
||||
|
||||
## Graph
|
||||
|
||||
Enabling a [graph network](../configuration/graph) adds a semantic graph at index time as data is being vectorized. Vector embeddings are used to automatically create relationships in the graph. Relationships can also be manually specified at index time.
|
||||
|
||||
```python
|
||||
# Manual relationships by id
|
||||
embeddings.index([{"id": "0", "text": "...", "relationships": ["2"]}])
|
||||
|
||||
# Manual relationships with additional edge attributes
|
||||
embeddings.index(["id": "0", "text": "...", "relationships": [
|
||||
{"id": "2", "type": "MEMBER_OF"}
|
||||
]])
|
||||
```
|
||||
|
||||
Additionally, graphs can be used for topic modeling. Dimensionality reduction with UMAP combined with HDBSCAN is a popular topic modeling method found in a number of libraries. txtai takes a different approach using community detection algorithms to build topic clusters.
|
||||
|
||||
This approach has the advantage of only having to vectorize data once. It also has the advantage of better topic precision given there isn't a dimensionality reduction operation (UMAP). Semantic graph examples are shown below.
|
||||
|
||||
Get a mapping of discovered topics to associated ids.
|
||||
|
||||
```python
|
||||
embeddings.graph.topics
|
||||
```
|
||||
|
||||
Show the most central nodes in the index.
|
||||
|
||||
```python
|
||||
embeddings.graph.centrality()
|
||||
```
|
||||
|
||||
Graphs are persisted alongside an embeddings index. Each save and load will also save and load the graph.
|
||||
|
||||
## Sparse vectors
|
||||
|
||||
Scoring instances can create a standalone [sparse keyword indexes](../configuration/general#keyword) (BM25, TF-IDF) and [sparse vector indexes](../configuration/general#sparse) (SPLADE). This enables [hybrid](../configuration/general/#hybrid) search when there is an available dense vector index.
|
||||
|
||||
The terms `sparse vector index`, `keyword index`, `terms index` and `scoring index` are used interchangeably throughout txtai's documentation.
|
||||
|
||||
See [this link](../../examples/#semantic-search) to learn more.
|
||||
|
||||
## Subindexes
|
||||
|
||||
An embeddings instance can optionally have associated [subindexes](../configuration/general/#indexes), which are also embeddings databases. This enables indexing additional fields, vector models and much more.
|
||||
|
||||
## Word vectors
|
||||
|
||||
When using [word vector backed models](../configuration/vectors#words) with scoring set, a separate call is required before calling `index` as follows:
|
||||
|
||||
```python
|
||||
embeddings.score(rows)
|
||||
embeddings.index(rows)
|
||||
```
|
||||
|
||||
Both calls are required to support generator-backed iteration of data with word vectors models.
|
||||
@@ -0,0 +1,20 @@
|
||||
# Methods
|
||||
|
||||
::: txtai.embeddings.Embeddings
|
||||
options:
|
||||
filters:
|
||||
- "!columns"
|
||||
- "!createann"
|
||||
- "!createcloud"
|
||||
- "!createdatabase"
|
||||
- "!creategraph"
|
||||
- "!createids"
|
||||
- "!createindexes"
|
||||
- "!createscoring"
|
||||
- "!checkarchive"
|
||||
- "!configure"
|
||||
- "!defaultallowed"
|
||||
- "!defaults"
|
||||
- "!initindex"
|
||||
- "!loadquery"
|
||||
- "!loadvectors"
|
||||
@@ -0,0 +1,285 @@
|
||||
# Query guide
|
||||
|
||||
|
||||
|
||||
|
||||
This section covers how to query data with txtai. The simplest way to search for data is building a natural language string with the desired content to find. txtai also supports querying with SQL. We'll cover both methods here.
|
||||
|
||||
## Natural language queries
|
||||
|
||||
In the simplest case, the query is text and the results are index text that is most similar to the query text.
|
||||
|
||||
```python
|
||||
embeddings.search("feel good story")
|
||||
embeddings.search("wildlife")
|
||||
```
|
||||
|
||||
The queries above [search](../methods#txtai.embeddings.base.Embeddings.search) the index for similarity matches on `feel good story` and `wildlife`. If content storage is enabled, a list of `{**query columns}` is returned. Otherwise, a list of `(id, score)` tuples are returned.
|
||||
|
||||
## SQL
|
||||
|
||||
txtai supports more complex queries with SQL. This is only supported if [content storage](../configuration/database#content) is enabled. txtai has a translation layer that analyzes input SQL statements and combines similarity results with content stored in a relational database.
|
||||
|
||||
SQL queries are run through `embeddings.search` like natural language queries but the examples below only show the SQL query for conciseness.
|
||||
|
||||
```python
|
||||
embeddings.search("SQL query")
|
||||
```
|
||||
|
||||
### Similar clause
|
||||
|
||||
The similar clause is a txtai function that enables similarity searches with SQL.
|
||||
|
||||
```sql
|
||||
SELECT id, text, score FROM txtai WHERE similar('feel good story')
|
||||
```
|
||||
|
||||
The similar clause takes the following arguments:
|
||||
|
||||
```sql
|
||||
similar("query", "number of candidates", "index", "weights")
|
||||
```
|
||||
|
||||
| Argument | Description |
|
||||
| --------------------- | ---------------------------------------|
|
||||
| query | natural language query to run |
|
||||
| number of candidates | number of candidate results to return |
|
||||
| index | target index name |
|
||||
| weights | hybrid score weights |
|
||||
|
||||
The txtai query layer joins results from two separate components, a relational store and a similarity index. With a similar clause, a similarity search is run and those ids are fed to the underlying database query.
|
||||
|
||||
The number of candidates should be larger than the desired number of results when applying additional filter clauses. This ensures that `limit` results are still returned after applying additional filters. If the number of candidates is not specified, it is defaulted as follows:
|
||||
|
||||
- For a single query filter clause, the default is the query limit
|
||||
- With multiple filtering clauses, the default is 10x the query limit
|
||||
|
||||
The index name is only applicable when [subindexes](../configuration/general/#indexes) are enabled. This specifies the index to use for the query.
|
||||
|
||||
Weights sets the hybrid score weights when an index has both a sparse and dense index.
|
||||
|
||||
### Dynamic columns
|
||||
|
||||
Content can be indexed in multiple ways when content storage is enabled. [Remember that input documents](../#index) take the form of `(id, data, tags)` tuples. If data is a string or binary content, it's indexed and searchable with `similar()` clauses.
|
||||
|
||||
If data is a dictionary, then all fields in the dictionary are stored and available via SQL. The `text` field or [field specified in the index configuration](../configuration/general/#columns) is indexed and searchable with `similar()` clauses.
|
||||
|
||||
For example:
|
||||
|
||||
```python
|
||||
embeddings.index([{"text": "text to index", "flag": True,
|
||||
"actiondate": "2022-01-01"}])
|
||||
```
|
||||
|
||||
With the above input data, queries can now have more complex filters.
|
||||
|
||||
```sql
|
||||
SELECT text, flag, actiondate FROM txtai WHERE similar('query') AND flag = 1
|
||||
AND actiondate >= '2022-01-01'
|
||||
```
|
||||
|
||||
txtai's query layer automatically detects columns and translates queries into a format that can be understood by the underlying database.
|
||||
|
||||
Nested dictionaries/JSON is supported and can be escaped with bracket statements.
|
||||
|
||||
```python
|
||||
embeddings.index([{"text": "text to index",
|
||||
"parent": {"child element": "abc"}}])
|
||||
```
|
||||
|
||||
```sql
|
||||
SELECT text FROM txtai WHERE [parent.child element] = 'abc'
|
||||
```
|
||||
|
||||
Note the bracket statement escaping the nested column with spaces in the name.
|
||||
|
||||
### Bind parameters
|
||||
|
||||
txtai has support for SQL bind parameters.
|
||||
|
||||
```python
|
||||
# Query with a bind parameter for similar clause
|
||||
query = "SELECT id, text, score FROM txtai WHERE similar(:x)"
|
||||
results = embeddings.search(query, parameters={"x": "feel good story"})
|
||||
|
||||
# Query with a bind parameter for column filter
|
||||
query = "SELECT text, flag, actiondate FROM txtai WHERE flag = :x"
|
||||
results = embeddings.search(query, parameters={"x": 1})
|
||||
```
|
||||
|
||||
### Aggregation queries
|
||||
|
||||
The goal of txtai's query language is to closely support all functions in the underlying database engine. The main challenge is ensuring dynamic columns are properly escaped into the engines native query function.
|
||||
|
||||
Aggregation query examples.
|
||||
|
||||
```sql
|
||||
SELECT count(*) FROM txtai WHERE similar('feel good story') AND score >= 0.15
|
||||
SELECT max(length(text)) FROM txtai WHERE similar('feel good story')
|
||||
AND score >= 0.15
|
||||
SELECT count(*), flag FROM txtai GROUP BY flag ORDER BY count(*) DESC
|
||||
```
|
||||
|
||||
## Binary objects
|
||||
|
||||
txtai has support for storing and retrieving binary objects. Binary objects can be retrieved as shown in the example below.
|
||||
|
||||
```python
|
||||
# Create embeddings index with content and object storage enabled
|
||||
embeddings = Embeddings(content=True, objects=True)
|
||||
|
||||
# Get an image
|
||||
request = open("demo.gif", "rb")
|
||||
|
||||
# Insert record
|
||||
embeddings.index([(
|
||||
"txtai",
|
||||
{"text": "txtai executes machine-learning workflows.",
|
||||
"object": request.read()}
|
||||
)])
|
||||
|
||||
# Query txtai and get associated object
|
||||
query = "SELECT object FROM txtai WHERE similar('machine learning') LIMIT 1"
|
||||
result = embeddings.search(query)[0]["object"]
|
||||
|
||||
# Query binary content with a bind parameter
|
||||
query = "SELECT object FROM txtai WHERE similar(:x) LIMIT 1"
|
||||
results = embeddings.search(query, parameters={"x": request.read()})
|
||||
```
|
||||
|
||||
## Custom SQL functions
|
||||
|
||||
Custom, user-defined SQL functions extend selection, filtering and ordering clauses with additional logic. For example, the following snippet defines a function that translates text using a translation pipeline.
|
||||
|
||||
```python
|
||||
# Translation pipeline
|
||||
translate = Translation()
|
||||
|
||||
# Create embeddings index
|
||||
embeddings = Embeddings(path="sentence-transformers/nli-mpnet-base-v2",
|
||||
content=True,
|
||||
functions=[translate]})
|
||||
|
||||
# Run a search using a custom SQL function
|
||||
embeddings.search("""
|
||||
SELECT
|
||||
text,
|
||||
translation(text, 'de', null) 'text (DE)',
|
||||
translation(text, 'es', null) 'text (ES)',
|
||||
translation(text, 'fr', null) 'text (FR)'
|
||||
FROM txtai WHERE similar('feel good story')
|
||||
LIMIT 1
|
||||
""")
|
||||
```
|
||||
|
||||
## Expressions
|
||||
|
||||
Expression shortcuts expand into more complex SQL snippets. This is useful for making SQL queries more concise. Indexing is also available on expressions as a performance improvement.
|
||||
|
||||
The following example indexes a json extraction field (`filepath`) and the length of each field.
|
||||
|
||||
```python
|
||||
# Create embeddings index
|
||||
embeddings = Embeddings(
|
||||
path="sentence-transformers/nli-mpnet-base-v2",
|
||||
content=True,
|
||||
expressions=[
|
||||
{"name": "filepath", "index": True},
|
||||
{"name": "textlength", "expression": "length(text)", "index": True}
|
||||
]
|
||||
)
|
||||
|
||||
embeddings.search("SELECT textlength, filepath FROM txtai LIMIT 1")
|
||||
```
|
||||
|
||||
## Query translation
|
||||
|
||||
Natural language queries with filters can be converted to txtai-compatible SQL statements with query translation. For example:
|
||||
|
||||
```python
|
||||
embeddings.search("feel good story since yesterday")
|
||||
```
|
||||
|
||||
can be converted to a SQL statement with a similar clause and date filter.
|
||||
|
||||
```sql
|
||||
select id, text, score from txtai where similar('feel good story') and
|
||||
entry >= date('now', '-1 day')
|
||||
```
|
||||
|
||||
This requires setting a [query translation model](../configuration/database#query). The default query translation model is [t5-small-txtsql](https://huggingface.co/NeuML/t5-small-txtsql) but this can easily be finetuned to handle different use cases.
|
||||
|
||||
## Hybrid search
|
||||
|
||||
When an embeddings database has both a sparse and dense index, both indexes will be queried and the results will be equally weighted unless otherwise specified.
|
||||
|
||||
```python
|
||||
embeddings.search("query", weights=0.5)
|
||||
embeddings.search(
|
||||
"SELECT id, text, score FROM txtai WHERE similar('query', 0.5)"
|
||||
)
|
||||
```
|
||||
|
||||
## Graph search
|
||||
|
||||
If an embeddings database has an associated graph network, graph searches can be run. The search syntax below uses [openCypher](https://github.com/opencypher/openCypher). Follow the preceding link to learn more about this syntax.
|
||||
|
||||
Additionally, standard embeddings searches can be returned as graphs.
|
||||
|
||||
```python
|
||||
# Find all paths between id: 0 and id: 5 between 1 and 3 hops away
|
||||
embeddings.search("""
|
||||
MATCH P=({id: 0})-[*1..3]->({id: 5})
|
||||
RETURN P
|
||||
""")
|
||||
|
||||
# Find related nodes for query matches
|
||||
embeddings.search("""
|
||||
MATCH P=(A)-[]->(B)
|
||||
WHERE SIMILAR(A, "query")
|
||||
RETURN B
|
||||
ORDER BY A.score DESC
|
||||
LIMIT 10
|
||||
""")
|
||||
|
||||
# Standard embeddings search as graph
|
||||
embeddings.search("query", graph=True)
|
||||
```
|
||||
|
||||
## Subindexes
|
||||
|
||||
Subindexes can be queried as follows:
|
||||
|
||||
```python
|
||||
# Build index with subindexes
|
||||
embeddings = Embeddings(
|
||||
content=True,
|
||||
defaults=False,
|
||||
indexes={
|
||||
"keyword": {
|
||||
"keyword": True
|
||||
},
|
||||
"dense":{
|
||||
"dense": True
|
||||
}
|
||||
}
|
||||
)
|
||||
embeddings.index(stream())
|
||||
|
||||
# Query with index parameter
|
||||
embeddings.search("query", index="keyword")
|
||||
|
||||
# Specify with SQL
|
||||
embeddings.search("""
|
||||
SELECT id, text, score FROM txtai
|
||||
WHERE similar('query', 'keyword')
|
||||
""")
|
||||
```
|
||||
|
||||
## Combined index architecture
|
||||
|
||||
txtai has multiple storage and indexing components. Content is stored in an underlying database along with an approximate nearest neighbor (ANN) index, keyword index and graph network. These components combine to deliver similarity search alongside traditional structured search.
|
||||
|
||||
The ANN index stores ids and vectors for each input element. When a natural language query is run, the query is translated into a vector and a similarity query finds the best matching ids. When a database is added into the mix, an additional step is executed. This step takes those ids and effectively inserts them as part of the underlying database query. The same steps apply with keyword indexes except a term frequency index is used to find the best matching ids.
|
||||
|
||||
Dynamic columns are supported via the underlying engine. For SQLite, data is stored as JSON and dynamic columns are converted into `json_extract` clauses. Client-server databases are supported via [SQLAlchemy](https://docs.sqlalchemy.org/en/20/dialects/) and dynamic columns are supported provided the underlying engine has [JSON](https://docs.sqlalchemy.org/en/20/core/type_basics.html#sqlalchemy.types.JSON) support.
|
||||
Reference in New Issue
Block a user