wehub-resource-sync
257 lines
9.5 KiB
Plaintext
257 lines
9.5 KiB
Plaintext
---
|
|
id: chroma
|
|
title: Chroma
|
|
sidebar_label: Chroma
|
|
---
|
|
|
|
## Quick Summary
|
|
|
|
**Chroma** is one of the most popular open-source AI application databases, and supports many retrieval features such as embeddings storage, vector search, document storage, metadata filtering, and multi-modal retrieval.
|
|
|
|
DeepEval allows you to easily evaluate and optimize your Chroma retriever by **tuning hyperparameters** like `n_results` (more commonly known as top-K) and the `embedding model` used in your Chroma retrieval pipeline.
|
|
|
|
:::caution
|
|
Chroma is not only an optional retriever you can evaluate, it is also a **required dependency** for the `deepeval.synthesizer.generate_goldens_from_docs()` method.
|
|
This method uses Chroma as its built-in backend for chunk storage and retrieval during context construction. If you plan to generate goldens from documents, make sure to install `chromadb`:
|
|
:::
|
|
|
|
:::info
|
|
To get started, install Chroma through the CLI using the following command:
|
|
|
|
```
|
|
pip install chromadb
|
|
```
|
|
|
|
:::
|
|
|
|
To learn more about using Chroma for your RAG pipeline, [visit this page](https://www.trychroma.com/). The diagram below illustrates how you can utilize Chroma as the entire retrieval pipeline for your LLM application.
|
|
|
|
<div
|
|
style={{
|
|
display: "flex",
|
|
alignItems: "center",
|
|
justifyContent: "center",
|
|
flexDirection: "column",
|
|
}}
|
|
>
|
|
<ImageDisplayer src="https://www.trychroma.com/_next/static/media/computer.fcd1bd54.svg" />
|
|
<div
|
|
style={{
|
|
fontSize: "13px",
|
|
}}
|
|
>
|
|
Source: Chroma
|
|
</div>
|
|
</div>
|
|
|
|
## Setup Chroma
|
|
|
|
To get started with **Chroma**, initialize a persistent client and create a collection to store your documents. The collection acts as a vector database for storing and retrieving embeddings, while the persistent client ensures data is retained across sessions.
|
|
|
|
```python
|
|
import chromadb
|
|
|
|
# Initialize Chroma client
|
|
client = chromadb.PersistentClient(path="./chroma_db")
|
|
|
|
# Create or load a collection
|
|
collection = client.get_or_create_collection(name="rag_documents")
|
|
```
|
|
|
|
Next, define an **embedding model** (we'll use `sentence_transformers`) to convert document chunks into vectors before adding them to your Chroma collection, along with the document chunks as metadata.
|
|
|
|
```python
|
|
...
|
|
|
|
# Load an embedding model
|
|
from sentence_transformers import SentenceTransformer
|
|
model = SentenceTransformer("all-MiniLM-L6-v2")
|
|
|
|
# Example document chunks
|
|
document_chunks = [
|
|
"Chroma is an open-source vector database for efficient embedding retrieval.",
|
|
"It enables fast semantic search using vector similarity.",
|
|
"Chroma retrieves relevant data with cosine similarity.",
|
|
...
|
|
]
|
|
|
|
# Store chunks with embeddings in Chroma
|
|
for i, chunk in enumerate(document_chunks):
|
|
embedding = model.encode(chunk).tolist() # Convert text to vector
|
|
collection.add(
|
|
ids=[str(i)], # Unique ID for each document
|
|
embeddings=[embedding], # Vector representation
|
|
metadatas=[{"text": chunk}] # Store original text as metadata
|
|
)
|
|
```
|
|
|
|
You'll be querying from this Chroma collection during generation to retrieve relevant contexts based on the user `input`, before passing them along with your input into your LLM's prompt template.
|
|
|
|
:::note
|
|
By default, Chroma utilizes `cosine similarity` to find similar chunks.
|
|
:::
|
|
|
|
## Evaluating Chroma Retrieval
|
|
|
|
To evaluate your Chroma retriever, you'll first need to prepare an `input` query and generate a response from your RAG pipeline in order to create an `LLMTestCase`. You'll also need to extract the contexts retrieved from your Chroma collection during generation and prepare the expected LLM response to complete the `LLMTestCase`.
|
|
|
|
:::information
|
|
By default, `input` and `actual_output` are required for all metrics. However, `retrieval_context`, `context`, and `expected_output` are optional, and different metrics may or may not require additional parameters. To check the specific requirements, [visit the metrics section](/docs/metrics-introduction).
|
|
:::
|
|
|
|
After you've prepared your `LLMTestCase`, evaluating your Chroma retriever is as easy passing the test case along with your selection of metrics into DeepEval's `evaluate` function.
|
|
|
|
### Preparing your Test Case
|
|
|
|
To prepare our test case, we'll be using `"How does Chroma work?"` as our input. Before generating a response from your RAG pipeline, you'll first need to retrieve the relevant context using a `search` function. Our `search` function in the example below first embeds the input query before retrieving the top three most relevant text chunks (`n_results=3`) from our chroma collection.
|
|
|
|
```python
|
|
...
|
|
|
|
def search(query):
|
|
query_embedding = model.encode(query).tolist()
|
|
|
|
res = collection.query(
|
|
query_embeddings=[query_embedding],
|
|
n_results=3 # Retrieve top-K matches
|
|
)
|
|
|
|
return res["metadatas"][0][0]["text"] if res["metadatas"][0] else None
|
|
|
|
query = "How does Chroma work?"
|
|
retrieval_context = search(query)
|
|
```
|
|
|
|
Next, we'll pass the retrieved context from our Chroma collection into the LLM's prompt template to generate the final response.
|
|
|
|
```python
|
|
...
|
|
|
|
prompt = """
|
|
Answer the user question based on the supporting context.
|
|
|
|
User Question:
|
|
{input}
|
|
|
|
Supporting Context:
|
|
{retrieval_context}
|
|
"""
|
|
|
|
actual_output = generate(prompt) # Replace with your LLM function
|
|
print(actual_output)
|
|
print(expected_output)
|
|
```
|
|
|
|
Printing the `actual_output` generated by our RAG pipeline yields the following example:
|
|
|
|
```
|
|
Chroma is a lightweight vector database designed for AI applications, enabling fast semantic retrieval.
|
|
```
|
|
|
|
Let's compare this to the `expected_output` we've prepared:
|
|
|
|
```
|
|
Chroma is an open-source vector database that enables fast retrieval using cosine similarity.
|
|
```
|
|
|
|
With all the elements ready, we'll create an `LLMTestCase` by providing the input and expected output, along with the actual output and retrieved context.
|
|
|
|
```python
|
|
from deepeval.test_case import LLMTestCase
|
|
|
|
...
|
|
|
|
test_case = LLMTestCase(
|
|
input=input,
|
|
actual_output=actual_output,
|
|
retrieval_context=retrieval_context,
|
|
expected_output=expected_output
|
|
)
|
|
```
|
|
|
|
### Running Evaluations
|
|
|
|
To begin running evaluations, we'll need to define metrics relevant to our Chroma retriever. These include `ContextualRecallMetric`, `ContextualPrecisionMetric`, and `ContextualRelevancyMetric`, which specifically evaluate RAG retrievers.
|
|
|
|
:::tip
|
|
To learn more about how these metrics are calculated and why they're relevant to retrievers, visit the [individual metric pages](/docs/metrics-contextual-precision).
|
|
:::
|
|
|
|
```python
|
|
from deepeval.metrics import (
|
|
ContextualPrecisionMetric,
|
|
ContextualRecallMetric,
|
|
ContextualRelevancyMetric,
|
|
)
|
|
|
|
contextual_precision = ContextualPrecisionMetric()
|
|
contextual_recall = ContextualRecallMetric(),
|
|
contextual_relevancy = ContextualRelevancyMetric()
|
|
```
|
|
|
|
To run evaluations, simply pass the prepared test case you've prepared into the `evaluate` function, along with the retriever metrics you defined.
|
|
|
|
```
|
|
from deepeval import evaluate
|
|
|
|
...
|
|
|
|
evaluate(
|
|
[test_case],
|
|
metrics=[contextual_recall, contextual_precision, contextual_relevancy]
|
|
)
|
|
```
|
|
|
|
## Improving Chroma Retrieval
|
|
|
|
Hypothetically, we've run multiple inputs and prepared several test cases, consistently observing that the `Contextual Relevancy` score is below the required threshold.
|
|
|
|
| <div style={{width: "350px"}}>Inputs</div> | <div style={{width: "250px"}}>Contextual Relevancy Score</div> | <div style={{width: "250px"}}>Contextual Recall Score</div> |
|
|
| ------------------------------------------ | -------------------------------------------------------------- | ----------------------------------------------------------- |
|
|
| "How does Chroma work?" | 0.45 | 0.85 |
|
|
| "What is the retrieval process in Chroma?" | 0.43 | 0.92 |
|
|
| "Explain Chroma's vector database." | 0.55 | 0.67 |
|
|
|
|
This suggests that you may need to adjust the length of each document or tweak `n_results` to retrieve more relevant contexts from your Chroma collection. This is because Contextual Relevancy evaluates both the **retrieved text chunks and the top-K selection**.
|
|
|
|
:::tip
|
|
If you're curious about which metrics evaluate which specific retrieval parameters, [check out this guide](/guides/guides-rag-evaluation).
|
|
:::
|
|
|
|
Depending on the failing scores in your retriever, you'll want to experiment with different parameters (e.g., `n_results`, `embedding model`, etc.) in your Chroma retrieval pipeline until you're satisfied with the results. This can be as simple as writing a for loop to run evaluations many times:
|
|
|
|
```python
|
|
...
|
|
|
|
def search(query, n_results):
|
|
query_embedding = model.encode(query).tolist()
|
|
|
|
res = collection.query(
|
|
query_embeddings=[query_embedding],
|
|
n_results=n_results # Retrieve top-K matches
|
|
)
|
|
|
|
return res["metadatas"][0][0]["text"] if res["metadatas"][0] else None
|
|
|
|
|
|
# Define input and expected output
|
|
...
|
|
|
|
# Iterate over different top-K values
|
|
for top_k in [3, 5, 7]:
|
|
retrieval_context = search(input_query, top_k)
|
|
|
|
# Define test case
|
|
...
|
|
|
|
# Evaluate the retrieval quality
|
|
evaluate(
|
|
[test_case],
|
|
metrics=[contextual_recall, contextual_precision, contextual_relevancy]
|
|
)
|
|
```
|
|
|
|
:::note
|
|
If you need a systematic way to analyze your retriever and compare the effects of changing chroma hyperparameters side by side, you'll want to [log in to Confident AI](https://www.confident-ai.com/).
|
|
:::
|