wehub-resource-sync
2cab53bc94
Test Vector Database Adaptors / Test MCP Vector DB Tools (push) Has been cancelled
Tests / Code Quality (Ruff & Mypy) (push) Has been cancelled
Tests / Fast Unit Tests (parallel) (macos-latest, 3.11) (push) Has been cancelled
Tests / Fast Unit Tests (parallel) (macos-latest, 3.12) (push) Has been cancelled
Tests / Tests (push) Has been cancelled
Docker Publish / Build and Push Docker Images (map[description:Skill Seekers CLI - Convert documentation to AI skills dockerfile:Dockerfile name:skill-seekers]) (push) Has been cancelled
Docker Publish / Build and Push Docker Images (map[description:Skill Seekers MCP Server - 25 tools for AI assistants dockerfile:Dockerfile.mcp name:skill-seekers-mcp]) (push) Has been cancelled
Docker Publish / Test Docker Images (push) Has been cancelled
Tests / Fast Unit Tests (parallel) (ubuntu-latest, 3.10) (push) Has been cancelled
Tests / Fast Unit Tests (parallel) (ubuntu-latest, 3.11) (push) Has been cancelled
Tests / Fast Unit Tests (parallel) (ubuntu-latest, 3.12) (push) Has been cancelled
Tests / Serial / Integration / E2E Tests (push) Has been cancelled
Tests / MCP Server Tests (push) Has been cancelled
Test Vector Database Adaptors / Test chroma Adaptor (push) Has been cancelled
Test Vector Database Adaptors / Test faiss Adaptor (push) Has been cancelled
Test Vector Database Adaptors / Test qdrant Adaptor (push) Has been cancelled
Test Vector Database Adaptors / Test weaviate Adaptor (push) Has been cancelled
96 lines
2.2 KiB
Markdown
96 lines
2.2 KiB
Markdown
# FAISS Vector Database Example
|
|
|
|
Facebook AI Similarity Search (FAISS) is a library for efficient similarity search of dense vectors. Perfect for large-scale semantic search.
|
|
|
|
## Quick Start
|
|
|
|
```bash
|
|
# 1. Install dependencies
|
|
pip install -r requirements.txt
|
|
|
|
# 2. Generate skill
|
|
python 1_generate_skill.py
|
|
|
|
# 3. Build FAISS index (requires OpenAI API key)
|
|
export OPENAI_API_KEY=sk-...
|
|
python 2_build_faiss_index.py
|
|
|
|
# 4. Query the index
|
|
python 3_query_example.py
|
|
```
|
|
|
|
## What's Different About FAISS?
|
|
|
|
- **No database server**: Pure Python library
|
|
- **Blazing fast**: Optimized C++ implementation
|
|
- **Scales to billions**: Efficient for massive datasets
|
|
- **Requires embeddings**: You must generate vectors (we use OpenAI)
|
|
|
|
## Key Features
|
|
|
|
### Generate Embeddings
|
|
|
|
FAISS doesn't generate embeddings - you must provide them:
|
|
|
|
```python
|
|
from openai import OpenAI
|
|
client = OpenAI()
|
|
|
|
# Generate embedding
|
|
response = client.embeddings.create(
|
|
model="text-embedding-ada-002",
|
|
input="Your text here"
|
|
)
|
|
embedding = response.data[0].embedding # 1536-dim vector
|
|
```
|
|
|
|
### Build Index
|
|
|
|
```python
|
|
import faiss
|
|
import numpy as np
|
|
|
|
# Create index (L2 distance)
|
|
dimension = 1536 # OpenAI ada-002
|
|
index = faiss.IndexFlatL2(dimension)
|
|
|
|
# Add vectors
|
|
vectors = np.array(embeddings).astype('float32')
|
|
index.add(vectors)
|
|
|
|
# Save to disk
|
|
faiss.write_index(index, "skill.index")
|
|
```
|
|
|
|
### Search
|
|
|
|
```python
|
|
# Load index
|
|
index = faiss.read_index("skill.index")
|
|
|
|
# Query (returns distances + indices)
|
|
distances, indices = index.search(query_vector, k=5)
|
|
```
|
|
|
|
## Cost Estimate
|
|
|
|
OpenAI embeddings: ~$0.10 per 1M tokens
|
|
- 20 documents (~10K tokens): < $0.001
|
|
- 1000 documents (~500K tokens): ~$0.05
|
|
|
|
## Files Structure
|
|
|
|
- `1_generate_skill.py` - Package for FAISS
|
|
- `2_build_faiss_index.py` - Generate embeddings & build index
|
|
- `3_query_example.py` - Search queries
|
|
|
|
## Resources
|
|
|
|
- **FAISS GitHub**: https://github.com/facebookresearch/faiss
|
|
- **FAISS Wiki**: https://github.com/facebookresearch/faiss/wiki
|
|
- **OpenAI Embeddings**: https://platform.openai.com/docs/guides/embeddings
|
|
|
|
---
|
|
|
|
**Note**: FAISS is best for advanced users who need maximum performance at scale. For simpler use cases, try ChromaDB or Weaviate.
|