Update README.md

Added the correct discord link

Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "memvid-core"
-version = "2.0.137"
+version = "2.0.140"
 edition = "2024"
 rust-version = "1.85.0"
 license = "Apache-2.0"

README.md

@@ -38,12 +38,10 @@
   <a href="https://github.com/memvid/memvid/stargazers"><img src="https://img.shields.io/github/stars/memvid/memvid?style=flat-square&logo=github" alt="Stars" /></a>
   <a href="https://github.com/memvid/memvid/network/members"><img src="https://img.shields.io/github/forks/memvid/memvid?style=flat-square&logo=github" alt="Forks" /></a>
   <a href="https://github.com/memvid/memvid/issues"><img src="https://img.shields.io/github/issues/memvid/memvid?style=flat-square&logo=github" alt="Issues" /></a>
-  <a href="https://discord.gg/2mynS7fcK7"><img src="https://img.shields.io/discord/1442910055233224745?style=flat-square&logo=discord&label=discord" alt="Discord" /></a>
+  <a href="https://discord.gg/7RUve6Zrrv"><img src="https://img.shields.io/discord/1442910055233224745?style=flat-square&logo=discord&label=discord" alt="Discord" /></a>
 </p>
 
 
-
-
 ## Benchmark Highlights
 
 **🚀 Higher accuracy than any other memory system :** +35% SOTA on LoCoMo, best-in-class long-horizon conversational recall & reasoning

docs/i18n/README.es.md

@@ -58,6 +58,18 @@
 <h2 align="center">⭐️ Deja una STAR para apoyar el proyecto ⭐️</h2>
 </p>
 
+## Lo más destacado de los benchmarks
+
+**🚀 Mayor precisión que cualquier otro sistema de memoria:** +35% SOTA en LoCoMo, con recall y razonamiento conversacional de largo horizonte de primer nivel.
+
+**🧠 Mejor razonamiento multi-hop y temporal:** +76% en multi-hop y +56% en temporal frente al promedio de la industria.
+
+**⚡ Latencia ultra baja a escala:** 0.025 ms P50 y 0.075 ms P99, con 1,372× más throughput que los enfoques estándar.
+
+**🔬 Benchmarks totalmente reproducibles:** LoCoMo (10 conversaciones de ~26K tokens), evaluación open source y LLM-as-Judge.
+
+---
+
 ## ¿Qué es Memvid?
 
 Memvid es un sistema de memoria portable para IA que empaqueta tus datos, embeddings, estructura de búsqueda y metadatos en un solo archivo.
@@ -152,16 +164,18 @@ memvid-core = "2.0"
 
 ### Feature Flags
 
-| Feature             | Description                                    |
-| ------------------- | ---------------------------------------------- |
-| `lex`               | Full-text search with BM25 ranking (Tantivy)   |
-| `pdf_extract`       | Pure Rust PDF text extraction                  |
-| `vec`               | Vector similarity search (HNSW + ONNX)         |
-| `clip`              | CLIP visual embeddings for image search        |
-| `whisper`           | Audio transcription with Whisper               |
-| `temporal_track`    | Natural language date parsing ("last Tuesday") |
-| `parallel_segments` | Multi-threaded ingestion                       |
-| `encryption`        | Password-based encryption capsules (.mv2e)     |
+| Feature             | Descripción                                                        |
+| ------------------- | ------------------------------------------------------------------ |
+| `lex`               | Búsqueda full-text con ranking BM25 (Tantivy)                      |
+| `pdf_extract`       | Extracción de texto PDF 100% en Rust                               |
+| `vec`               | Búsqueda por similitud vectorial (HNSW + embeddings locales vía ONNX) |
+| `clip`              | Embeddings visuales CLIP para búsqueda de imágenes                 |
+| `whisper`           | Transcripción de audio con Whisper                                 |
+| `api_embed`         | Embeddings en la nube mediante API (OpenAI)                        |
+| `temporal_track`    | Interpretación de fechas en lenguaje natural ("el martes pasado")  |
+| `parallel_segments` | Ingesta multi-hilo                                                 |
+| `encryption`        | Cápsulas cifradas con contraseña (.mv2e)                           |
+| `symspell_cleanup`  | Reparación robusta de texto PDF (corrige "emp lo yee" -> "employee") |
 
 Activa las features según lo necesites:
 
@@ -305,6 +319,137 @@ cargo run --example test_whisper --features whisper
 
 ---
 
+## Modelos de embeddings de texto
+
+La feature `vec` incluye soporte para embeddings de texto locales usando modelos ONNX. Antes de usar embeddings locales, necesitas descargar manualmente los archivos del modelo.
+
+### Inicio rápido: BGE-small (recomendado)
+
+Descarga el modelo BGE-small por defecto (384 dimensiones, rápido y eficiente):
+
+```bash
+mkdir -p ~/.cache/memvid/text-models
+
+# Descargar modelo ONNX
+curl -L 'https://huggingface.co/BAAI/bge-small-en-v1.5/resolve/main/onnx/model.onnx' \
+  -o ~/.cache/memvid/text-models/bge-small-en-v1.5.onnx
+
+# Descargar tokenizer
+curl -L 'https://huggingface.co/BAAI/bge-small-en-v1.5/resolve/main/tokenizer.json' \
+  -o ~/.cache/memvid/text-models/bge-small-en-v1.5_tokenizer.json
+```
+
+### Modelos disponibles
+
+| Modelo                  | Dimensiones | Tamaño | Mejor para            |
+| ----------------------- | ----------- | ------ | --------------------- |
+| `bge-small-en-v1.5`     | 384         | ~120MB | Opción por defecto, rápido |
+| `bge-base-en-v1.5`      | 768         | ~420MB | Mejor calidad         |
+| `nomic-embed-text-v1.5` | 768         | ~530MB | Tareas versátiles     |
+| `gte-large`             | 1024        | ~1.3GB | Máxima calidad        |
+
+### Otros modelos
+
+**BGE-base** (768 dimensiones):
+```bash
+curl -L 'https://huggingface.co/BAAI/bge-base-en-v1.5/resolve/main/onnx/model.onnx' \
+  -o ~/.cache/memvid/text-models/bge-base-en-v1.5.onnx
+curl -L 'https://huggingface.co/BAAI/bge-base-en-v1.5/resolve/main/tokenizer.json' \
+  -o ~/.cache/memvid/text-models/bge-base-en-v1.5_tokenizer.json
+```
+
+**Nomic** (768 dimensiones):
+```bash
+curl -L 'https://huggingface.co/nomic-ai/nomic-embed-text-v1.5/resolve/main/onnx/model.onnx' \
+  -o ~/.cache/memvid/text-models/nomic-embed-text-v1.5.onnx
+curl -L 'https://huggingface.co/nomic-ai/nomic-embed-text-v1.5/resolve/main/tokenizer.json' \
+  -o ~/.cache/memvid/text-models/nomic-embed-text-v1.5_tokenizer.json
+```
+
+**GTE-large** (1024 dimensiones):
+```bash
+curl -L 'https://huggingface.co/thenlper/gte-large/resolve/main/onnx/model.onnx' \
+  -o ~/.cache/memvid/text-models/gte-large.onnx
+curl -L 'https://huggingface.co/thenlper/gte-large/resolve/main/tokenizer.json' \
+  -o ~/.cache/memvid/text-models/gte-large_tokenizer.json
+```
+
+### Uso en código
+
+```rust
+use memvid_core::text_embed::{LocalTextEmbedder, TextEmbedConfig};
+use memvid_core::types::embedding::EmbeddingProvider;
+
+// Usar el modelo por defecto (BGE-small)
+let config = TextEmbedConfig::default();
+let embedder = LocalTextEmbedder::new(config)?;
+
+let embedding = embedder.embed_text("hello world")?;
+assert_eq!(embedding.len(), 384);
+
+// Usar un modelo distinto
+let config = TextEmbedConfig::bge_base();
+let embedder = LocalTextEmbedder::new(config)?;
+```
+
+Consulta `examples/text_embedding.rs` para ver un ejemplo completo con cálculo de similitud y ranking de búsqueda.
+
+### Consistencia del modelo
+
+Para evitar mezclar modelos por accidente, por ejemplo consultar un índice BGE-small con embeddings de OpenAI, puedes asociar explícitamente tu instancia de Memvid a un nombre de modelo:
+
+```rust
+// Vincula el índice a un modelo concreto.
+// Si el índice ya fue creado con otro modelo, devolverá un error.
+mem.set_vec_model("bge-small-en-v1.5")?;
+```
+
+Esta vinculación es persistente. Una vez definida, cualquier intento futuro de usar otro nombre de modelo fallará de inmediato con un error `ModelMismatch`.
+
+---
+
+## Embeddings por API (OpenAI)
+
+La feature `api_embed` habilita la generación de embeddings en la nube usando la API de OpenAI.
+
+### Configuración
+
+Define tu clave de API de OpenAI:
+
+```bash
+export OPENAI_API_KEY="sk-..."
+```
+
+### Uso
+
+```rust
+use memvid_core::api_embed::{OpenAIConfig, OpenAIEmbedder};
+use memvid_core::types::embedding::EmbeddingProvider;
+
+// Usar el modelo por defecto (text-embedding-3-small)
+let config = OpenAIConfig::default();
+let embedder = OpenAIEmbedder::new(config)?;
+
+let embedding = embedder.embed_text("hello world")?;
+assert_eq!(embedding.len(), 1536);
+
+// Usar un modelo de mayor calidad
+let config = OpenAIConfig::large();  // text-embedding-3-large (3072 dims)
+let embedder = OpenAIEmbedder::new(config)?;
+```
+
+### Modelos disponibles
+
+| Modelo                   | Dimensiones | Mejor para                       |
+| ------------------------ | ----------- | -------------------------------- |
+| `text-embedding-3-small` | 1536        | Por defecto, más rápido y económico |
+| `text-embedding-3-large` | 3072        | Máxima calidad                   |
+| `text-embedding-ada-002` | 1536        | Modelo heredado                  |
+
+Consulta `examples/openai_embedding.rs` para ver un ejemplo completo.
+
+---
+
 ## Formato de archivo
 
 Todo vive en un único archivo `.mv2`:
@@ -342,7 +487,14 @@ Email: contact@memvid.com
 
 ---
 
+> **Memvid v1 (memoria basada en QR) está obsoleto**
+>
+> Si estás viendo referencias a códigos QR, estás usando información desactualizada.
+>
+> Consulta: https://docs.memvid.com/memvid-v1-deprecation
+
+---
+
 ## Licencia
 
 Apache License 2.0 — consulta el archivo [LICENSE](LICENSE) para más detalles.
-

src/memvid/ask.rs

@@ -576,6 +576,7 @@ impl Memvid {
                     snippet_chars: request.snippet_chars,
                     cursor: search_request.cursor.clone(),
                 },
+                stale_index_skips: 0,
             });
         }
 
@@ -659,6 +660,7 @@ impl Memvid {
                 snippet_chars: request.snippet_chars,
                 cursor: search_request.cursor.clone(),
             },
+            stale_index_skips: 0,
         })
     }
 }

src/memvid/mutation.rs

@@ -429,6 +429,7 @@ impl Memvid {
         let original_data_end = self.data_end;
         let original_generation = self.generation;
         let original_dirty = self.dirty;
+        let original_lex_enabled = self.lex_enabled;
         #[cfg(feature = "lex")]
         let original_tantivy_dirty = self.tantivy_dirty;
 
@@ -462,6 +463,7 @@ impl Memvid {
                         self.data_end = original_data_end;
                         self.generation = original_generation;
                         self.dirty = original_dirty;
+                        self.lex_enabled = original_lex_enabled;
                         #[cfg(feature = "lex")]
                         {
                             self.tantivy_dirty = original_tantivy_dirty;
@@ -483,6 +485,7 @@ impl Memvid {
                 self.data_end = original_data_end;
                 self.generation = original_generation;
                 self.dirty = original_dirty;
+                self.lex_enabled = original_lex_enabled;
                 #[cfg(feature = "lex")]
                 {
                     self.tantivy_dirty = original_tantivy_dirty;
@@ -604,9 +607,7 @@ impl Memvid {
 
         self.shift_data_for_wal_growth(delta)?;
         self.header.wal_size = new_size;
-        self.header.footer_offset = self.header.footer_offset.saturating_add(delta);
-        self.data_end = self.data_end.saturating_add(delta);
-        self.adjust_offsets_after_wal_growth(delta);
+        self.apply_wal_growth_offsets(delta);
 
         let catalog_end = self.catalog_data_end();
         self.header.footer_offset = catalog_end
@@ -656,6 +657,24 @@ impl Memvid {
         Ok(())
     }
 
+    fn apply_wal_growth_offsets(&mut self, delta: u64) {
+        if delta == 0 {
+            return;
+        }
+
+        self.header.footer_offset = self.header.footer_offset.saturating_add(delta);
+        self.data_end = self.data_end.saturating_add(delta);
+        // `cached_payload_end` mirrors `data_end` / footer positioning. After
+        // `shift_data_for_wal_growth` every byte past the old WAL boundary
+        // moved right by `delta`, so the cached payload boundary must also
+        // advance. Forgetting this caused `rebuild_indexes` (which seeks to
+        // `payload_region_end()`) to write embedded indexes back into the
+        // grown portion of the WAL region, corrupting WAL record payloads.
+        // See https://github.com/memvid/memvid/issues/230.
+        self.cached_payload_end = self.cached_payload_end.saturating_add(delta);
+        self.adjust_offsets_after_wal_growth(delta);
+    }
+
     fn adjust_offsets_after_wal_growth(&mut self, delta: u64) {
         if delta == 0 {
             return;
@@ -798,9 +817,7 @@ impl Memvid {
 
         self.shift_data_for_wal_growth(delta)?;
         self.header.wal_size = target;
-        self.header.footer_offset = self.header.footer_offset.saturating_add(delta);
-        self.data_end = self.data_end.saturating_add(delta);
-        self.adjust_offsets_after_wal_growth(delta);
+        self.apply_wal_growth_offsets(delta);
 
         let catalog_end = self.catalog_data_end();
         self.header.footer_offset = catalog_end

src/memvid/search/api.rs

@@ -19,10 +19,17 @@ impl Memvid {
     pub fn enable_lex(&mut self) -> Result<()> {
         self.ensure_writable()?;
         if self.lex_enabled {
-            // If index exists on disk but not in memory, load it
             #[cfg(feature = "lex")]
-            if self.lex_index.is_none() && crate::memvid::lifecycle::has_lex_index(&self.toc) {
-                self.load_lex_index_from_manifest()?;
+            {
+                // If index exists on disk but not in memory, load it
+                if self.lex_index.is_none() && crate::memvid::lifecycle::has_lex_index(&self.toc) {
+                    self.load_lex_index_from_manifest()?;
+                }
+                // Ensure Tantivy engine is running even if lex was already enabled
+                // (e.g. create() set lex_enabled=true but tantivy may have been lost)
+                if self.tantivy.is_none() {
+                    self.init_tantivy()?;
+                }
             }
             return Ok(());
         }
@@ -353,6 +360,7 @@ impl Memvid {
                 context: build_context(&[]),
                 next_cursor: None,
                 engine: SearchEngineKind::Hybrid,
+                stale_index_skips: 0,
             });
         }
 
@@ -457,6 +465,7 @@ impl Memvid {
             context,
             next_cursor: None,
             engine: SearchEngineKind::Hybrid,
+            stale_index_skips: 0,
         })
     }
 

src/memvid/search/fallback.rs

@@ -39,18 +39,24 @@ pub(super) fn search_with_lex_fallback(
     let max_snippets_per_doc = request.top_k.max(1);
 
     let mut evaluated = Vec::new();
+    let mut stale_skips = 0u32;
     for matched in &matches {
         if let Some(filter) = candidate_filter {
             if !filter.contains(&matched.frame_id) {
                 continue;
             }
         }
-        let frame_meta = usize::try_from(matched.frame_id)
+        let Some(frame_meta) = usize::try_from(matched.frame_id)
             .ok()
             .and_then(|idx| memvid.toc.frames.get(idx))
-            .ok_or(MemvidError::InvalidTimeIndex {
-                reason: "frame id out of range".into(),
-            })?;
+        else {
+            tracing::warn!(
+                frame_id = matched.frame_id,
+                "skipping search hit with stale frame_id"
+            );
+            stale_skips = stale_skips.saturating_add(1);
+            continue;
+        };
         let content_lower = matched.content.to_ascii_lowercase();
         let ctx = EvaluationContext {
             frame: frame_meta,
@@ -86,14 +92,18 @@ pub(super) fn search_with_lex_fallback(
     let mut hits = Vec::new();
     let mut produced = 0usize;
     for (matched, slices) in evaluated {
-        let frame_meta = memvid
+        let Some(frame_meta) = memvid
             .toc
             .frames
             .get(usize::try_from(matched.frame_id).unwrap_or(usize::MAX))
             .cloned()
-            .ok_or(MemvidError::InvalidTimeIndex {
-                reason: "frame id out of range".into(),
-            })?;
+        else {
+            tracing::warn!(
+                frame_id = matched.frame_id,
+                "skipping stale frame_id in snippet assembly"
+            );
+            continue;
+        };
         let canonical = memvid.frame_content(&frame_meta)?;
         let canonical_limit = frame_meta.canonical_length.map_or_else(
             || canonical.len(),
@@ -200,6 +210,7 @@ pub(super) fn search_with_lex_fallback(
         context,
         next_cursor,
         engine: SearchEngineKind::LexFallback,
+        stale_index_skips: stale_skips,
     })
 }
 
@@ -250,6 +261,7 @@ pub(super) fn search_with_filters_only(
             context: build_context(&[]),
             next_cursor: None,
             engine: SearchEngineKind::LexFallback,
+            stale_index_skips: 0,
         });
     }
 
@@ -322,5 +334,6 @@ pub(super) fn search_with_filters_only(
         context,
         next_cursor,
         engine: SearchEngineKind::LexFallback,
+        stale_index_skips: 0,
     })
 }

src/memvid/search/helpers.rs

@@ -33,6 +33,7 @@ pub(super) fn empty_search_response(
         context: String::new(),
         next_cursor: None,
         engine,
+        stale_index_skips: 0,
     }
 }
 
@@ -344,13 +345,18 @@ pub(crate) fn attach_temporal_metadata(memvid: &mut Memvid, hits: &mut [SearchHi
 
                 let text = if mention_end > mention_start {
                     if !canonical_cache.contains_key(&frame_id) {
-                        let frame = memvid.toc.frames.get(frame_id as usize).cloned().ok_or(
-                            MemvidError::InvalidTimeIndex {
-                                reason: "frame id out of range".into(),
-                            },
-                        )?;
-                        let content = memvid.frame_content(&frame)?;
-                        canonical_cache.insert(frame_id, content);
+                        match memvid.toc.frames.get(frame_id as usize).cloned() {
+                            Some(frame) => {
+                                let content = memvid.frame_content(&frame)?;
+                                canonical_cache.insert(frame_id, content);
+                            }
+                            None => {
+                                tracing::warn!(
+                                    frame_id,
+                                    "skipping temporal text for stale frame_id"
+                                );
+                            }
+                        }
                     }
                     canonical_cache.get(&frame_id).and_then(|content| {
                         if mention_end <= content.len() {

src/memvid/search/mod.rs

@@ -48,6 +48,13 @@ impl Memvid {
             return Err(MemvidError::LexNotEnabled);
         }
 
+        // Lazy-init Tantivy if lex is enabled but engine is missing.
+        // This can happen when a wrapper re-enables lex on an already-enabled
+        // instance, or after a staging-lock rollback lost the engine reference.
+        if self.tantivy.is_none() {
+            self.init_tantivy()?;
+        }
+
         let start_time = Instant::now();
         // parse_query can return structured tokens; we only keep non-empty, lower-cased terms.
         let parsed = crate::search::parse_query(&request.query)?;

src/memvid/search/tantivy.rs

@@ -6,6 +6,7 @@ use super::helpers::attach_temporal_metadata;
 use super::helpers::{
     build_context, collect_token_occurrences, parse_cursor, timestamp_to_rfc3339,
 };
+use crate::Result;
 use crate::lex::compute_snippet_slices;
 use crate::memvid::frame::ChunkInfo;
 use crate::memvid::lifecycle::Memvid;
@@ -14,7 +15,6 @@ use crate::types::{
     FrameId, SearchEngineKind, SearchHit, SearchHitMetadata, SearchParams, SearchRequest,
     SearchResponse,
 };
-use crate::{MemvidError, Result};
 use log::warn;
 use std::collections::HashSet;
 use std::time::Instant;
@@ -119,15 +119,21 @@ pub(super) fn try_tantivy_search(
     let snippet_window = request.snippet_chars.max(80);
     let max_snippets_per_doc = request.top_k.max(1);
     let mut evaluated = Vec::new();
+    let mut stale_skips = 0u32;
     for hit in search_hits {
-        let frame_meta = memvid
+        let Some(frame_meta) = memvid
             .toc
             .frames
             .get(usize::try_from(hit.frame_id).unwrap_or(usize::MAX))
             .cloned()
-            .ok_or(MemvidError::InvalidTimeIndex {
-                reason: "frame id out of range".into(),
-            })?;
+        else {
+            tracing::warn!(
+                frame_id = hit.frame_id,
+                "skipping search hit with stale frame_id"
+            );
+            stale_skips = stale_skips.saturating_add(1);
+            continue;
+        };
         if let Some(uri_expected) = uri_filter {
             if !uri_matches(frame_meta.uri.as_deref(), uri_expected) {
                 continue;
@@ -274,14 +280,18 @@ pub(super) fn try_tantivy_search(
         if hits.len() == effective_top_k && produced >= offset {
             break;
         }
-        let frame_meta = memvid
+        let Some(frame_meta) = memvid
             .toc
             .frames
             .get(usize::try_from(hit.frame_id).unwrap_or(usize::MAX))
             .cloned()
-            .ok_or(MemvidError::InvalidTimeIndex {
-                reason: "frame id out of range".into(),
-            })?;
+        else {
+            tracing::warn!(
+                frame_id = hit.frame_id,
+                "skipping stale frame_id in snippet assembly"
+            );
+            continue;
+        };
         let uri = frame_meta
             .uri
             .clone()
@@ -373,6 +383,7 @@ pub(super) fn try_tantivy_search(
         context,
         next_cursor,
         engine: SearchEngineKind::Tantivy,
+        stale_index_skips: stale_skips,
     }))
 }
 

src/memvid/ticket.rs

@@ -117,6 +117,8 @@ impl Memvid {
             time_index_bytes,
             vector_count,
             clip_image_count,
+            lex_enabled: self.lex_enabled,
+            vec_enabled: self.vec_enabled,
         })
     }
 

src/memvid/timeline.rs

@@ -1,5 +1,6 @@
 //! Timeline assembly helpers for `Memvid`.
 
+use crate::Result;
 use crate::io::time_index::{TimeIndexEntry, read_track as time_index_read};
 use crate::memvid::lifecycle::Memvid;
 #[cfg(feature = "temporal_track")]
@@ -10,7 +11,6 @@ use crate::types::{
     SearchHitTemporal, SearchHitTemporalAnchor, SearchHitTemporalMention, TemporalFilter,
     TemporalTrack,
 };
-use crate::{MemvidError, Result};
 #[cfg(feature = "temporal_track")]
 use std::collections::HashSet;
 use std::num::NonZeroU64;
@@ -95,14 +95,18 @@ pub(crate) fn build_timeline(
     #[cfg(feature = "temporal_track")]
     let temporal_track_snapshot = memvid.temporal_track_ref()?.cloned();
     for entry in entries.into_iter().take(limit) {
-        let frame = memvid
+        let Some(frame) = memvid
             .toc
             .frames
             .get(usize::try_from(entry.frame_id).unwrap_or(usize::MAX))
-            .ok_or(MemvidError::InvalidTimeIndex {
-                reason: "frame id out of range".into(),
-            })?
-            .clone();
+            .cloned()
+        else {
+            tracing::warn!(
+                frame_id = entry.frame_id,
+                "skipping time index entry with out-of-range frame id"
+            );
+            continue;
+        };
         if frame.status != FrameStatus::Active {
             continue;
         }

src/reader/mod.rs

@@ -6,9 +6,9 @@ mod pdf;
 mod pptx;
 mod xls;
 mod xlsx;
-pub(crate) mod xlsx_chunker;
-pub(crate) mod xlsx_ooxml;
-pub(crate) mod xlsx_table_detect;
+pub mod xlsx_chunker;
+pub mod xlsx_ooxml;
+pub mod xlsx_table_detect;
 
 use serde_json::Value;
 

src/reader/xlsx_ooxml.rs

@@ -324,7 +324,7 @@ fn parse_workbook_sheet_names(xml: &str) -> Vec<String> {
             {
                 for attr in e.attributes().flatten() {
                     if attr.key.as_ref() == b"name" {
-                        if let Ok(val) = attr.unescape_value() {
+                        if let Ok(val) = attr.decode_and_unescape_value(&reader) {
                             names.push(val.to_string());
                         }
                     }
@@ -364,12 +364,12 @@ fn parse_styles_xml(xml: &str, metadata: &mut OoxmlMetadata) {
                         for attr in e.attributes().flatten() {
                             match attr.key.as_ref() {
                                 b"numFmtId" => {
-                                    if let Ok(v) = attr.unescape_value() {
+                                    if let Ok(v) = attr.decode_and_unescape_value(&reader) {
                                         fmt_id = v.parse::<u32>().ok();
                                     }
                                 }
                                 b"formatCode" => {
-                                    if let Ok(v) = attr.unescape_value() {
+                                    if let Ok(v) = attr.decode_and_unescape_value(&reader) {
                                         fmt_code = Some(v.to_string());
                                     }
                                 }
@@ -384,7 +384,7 @@ fn parse_styles_xml(xml: &str, metadata: &mut OoxmlMetadata) {
                         let mut num_fmt_id = 0u32;
                         for attr in e.attributes().flatten() {
                             if attr.key.as_ref() == b"numFmtId" {
-                                if let Ok(v) = attr.unescape_value() {
+                                if let Ok(v) = attr.decode_and_unescape_value(&reader) {
                                     num_fmt_id = v.parse::<u32>().unwrap_or(0);
                                 }
                             }
@@ -425,7 +425,7 @@ fn parse_merge_cells_xml(xml: &str) -> Vec<MergedRegion> {
             {
                 for attr in e.attributes().flatten() {
                     if attr.key.as_ref() == b"ref" {
-                        if let Ok(val) = attr.unescape_value() {
+                        if let Ok(val) = attr.decode_and_unescape_value(&reader) {
                             if let Some(((tr, lc), (br, rc))) = parse_range_ref(&val) {
                                 regions.push(MergedRegion {
                                     top_row: tr,

src/search/tantivy/engine.rs

@@ -12,8 +12,16 @@ use tantivy::{Index, IndexReader, Term, doc};
 use tempfile::TempDir;
 
 /// Tantivy-backed search index used when the `lex` feature is enabled.
+///
+/// Field order is load-bearing for `Drop`: Rust drops struct fields in
+/// declaration order, so `work_dir` (the temporary directory backing the
+/// index) **must remain the last field**. The `index`, `reader`, and
+/// `index_writer` all keep file handles and lock files open inside that
+/// directory; if the `TempDir` were dropped first, its directory removal
+/// would fail on platforms that refuse to delete files with open handles
+/// (notably Windows), silently leaking one working directory per discarded
+/// engine. See https://github.com/memvid/memvid/issues/215.
 pub struct TantivyEngine {
-    pub(super) work_dir: TempDir,
     pub(super) index: Index,
     pub(super) _schema: Schema,
     pub(super) content: Field,
@@ -26,6 +34,20 @@ pub struct TantivyEngine {
     pub(super) index_writer: Option<IndexWriter>,
     pub(super) reader: IndexReader,
     pub(super) tokenizer: Option<String>,
+    // MUST be the last field — see the type-level comment above.
+    pub(super) work_dir: TempDir,
+}
+
+impl Drop for TantivyEngine {
+    fn drop(&mut self) {
+        // Release the exclusive writer (and its `.tantivy-writer.lock`) before
+        // the `work_dir` TempDir is removed during normal field drop. Without
+        // this the writer lock can still be held when the directory removal
+        // runs, leaking the working directory on Windows. See issue #215.
+        if let Some(writer) = self.index_writer.take() {
+            drop(writer);
+        }
+    }
 }
 
 /// Search hit returned from Tantivy queries.

src/tests_lex_flag.rs

@@ -1,6 +1,7 @@
 #[cfg(test)]
 mod tests {
     use crate::{Memvid, PutOptions, SearchRequest, run_serial_test};
+    use std::sync::Mutex;
     use tempfile::NamedTempFile;
 
     #[test]
@@ -87,4 +88,129 @@ mod tests {
             }
         });
     }
+
+    /// Regression test for GitHub issue #201:
+    /// Lexical index not enabled when Memvid is wrapped in a Mutex.
+    /// The wrapper pattern acquires the lock, performs an operation, releases
+    /// the lock — mimicking the typical tokio::sync::Mutex usage in async code.
+    #[test]
+    #[cfg(not(target_os = "windows"))]
+    fn test_lex_works_through_mutex_wrapper() {
+        run_serial_test(|| {
+            let temp = NamedTempFile::new().unwrap();
+            let path = temp.path();
+
+            // Wrap Memvid in a Mutex, exactly like an async wrapper would
+            let wrapper = Mutex::new(Memvid::create(path).unwrap());
+
+            // Step 1: enable_lex while holding the lock, then release
+            {
+                let mut mem = wrapper.lock().unwrap();
+                mem.enable_lex().unwrap();
+            }
+
+            // Step 2: commit while holding the lock (separate acquisition)
+            {
+                let mut mem = wrapper.lock().unwrap();
+                mem.commit().unwrap();
+            }
+
+            // Step 3: put data while holding the lock
+            {
+                let mut mem = wrapper.lock().unwrap();
+                let opts = PutOptions::builder()
+                    .uri("mv2://test/login".to_string())
+                    .search_text("user clicked login button on the authentication page".to_string())
+                    .build();
+                mem.put_bytes_with_options(b"login event data", opts)
+                    .unwrap();
+            }
+
+            // Step 4: commit while holding the lock
+            {
+                let mut mem = wrapper.lock().unwrap();
+                mem.commit().unwrap();
+            }
+
+            // Step 5: search while holding the lock — this was failing in #201
+            {
+                let mut mem = wrapper.lock().unwrap();
+                let resp = mem
+                    .search(SearchRequest {
+                        query: "login".into(),
+                        top_k: 10,
+                        snippet_chars: 200,
+                        uri: None,
+                        scope: None,
+                        cursor: None,
+                        #[cfg(feature = "temporal_track")]
+                        temporal: None,
+                        as_of_frame: None,
+                        as_of_ts: None,
+                        no_sketch: false,
+                        acl_context: None,
+                        acl_enforcement_mode: crate::types::AclEnforcementMode::Audit,
+                    })
+                    .expect("search must succeed through mutex wrapper");
+
+                assert!(
+                    !resp.hits.is_empty(),
+                    "Should find the frame with 'login' in the message"
+                );
+            }
+
+            // Step 6: search_lex uses the legacy LexIndex, which may not be
+            // populated when only Tantivy is active. Verify it doesn't panic.
+            {
+                let mut mem = wrapper.lock().unwrap();
+                let _ = mem.search_lex("login", 10);
+                // Result may be Ok (if legacy index was built) or Err (if only Tantivy).
+                // The important thing is it doesn't panic.
+            }
+        });
+    }
+
+    /// Regression test for #201: enable_lex, put, commit, search — all in one lock scope.
+    #[test]
+    #[cfg(not(target_os = "windows"))]
+    fn test_lex_works_single_scope() {
+        run_serial_test(|| {
+            let temp = NamedTempFile::new().unwrap();
+            let path = temp.path();
+
+            let mut mem = Memvid::create(path).unwrap();
+            mem.enable_lex().unwrap();
+
+            let opts = PutOptions::builder()
+                .uri("mv2://test/login".to_string())
+                .search_text("user clicked login button on the authentication page".to_string())
+                .build();
+            mem.put_bytes_with_options(b"login event data", opts)
+                .unwrap();
+            mem.commit().unwrap();
+
+            let resp = mem
+                .search(SearchRequest {
+                    query: "login".into(),
+                    top_k: 10,
+                    snippet_chars: 200,
+                    uri: None,
+                    scope: None,
+                    cursor: None,
+                    #[cfg(feature = "temporal_track")]
+                    temporal: None,
+                    as_of_frame: None,
+                    as_of_ts: None,
+                    no_sketch: false,
+                    acl_context: None,
+                    acl_enforcement_mode: crate::types::AclEnforcementMode::Audit,
+                })
+                .expect("search must succeed");
+
+            assert!(
+                !resp.hits.is_empty(),
+                "Should find the frame with 'login' in the message"
+            );
+        });
+    }
 }

src/types/frame.rs

@@ -136,6 +136,12 @@ pub struct Stats {
     /// Number of CLIP visual embeddings (images/PDF pages)
     #[serde(default)]
     pub clip_image_count: u64,
+    /// Whether the lex (full-text) search engine is enabled at runtime.
+    #[serde(default)]
+    pub lex_enabled: bool,
+    /// Whether the vec (vector/semantic) search engine is enabled at runtime.
+    #[serde(default)]
+    pub vec_enabled: bool,
 }
 
 /// Entry returned by `timeline` queries, carrying a lightweight preview.

src/types/search.rs

@@ -190,4 +190,12 @@ pub struct SearchResponse {
     #[serde(default)]
     /// Engine responsible for the results.
     pub engine: SearchEngineKind,
+    /// Number of search hits skipped due to stale frame_ids in the index.
+    #[serde(default, skip_serializing_if = "is_zero")]
+    pub stale_index_skips: u32,
+}
+
+#[allow(clippy::trivially_copy_pass_by_ref)]
+const fn is_zero(v: &u32) -> bool {
+    *v == 0
 }

tests/mutation.rs

@@ -3,11 +3,23 @@
 
 use memvid_core::{
     EmbeddingIdentitySummary, MEMVID_EMBEDDING_MODEL_KEY, MEMVID_EMBEDDING_PROVIDER_KEY, Memvid,
-    MemvidError, PutOptions, TimelineQuery,
+    MemvidError, PutOptions, TimelineQuery, constants::HEADER_SIZE, io::header::HeaderCodec,
 };
+use std::fs::File;
+use std::io::Read;
 use std::num::NonZeroU64;
+use std::path::Path;
 use tempfile::TempDir;
 
+fn read_wal_size(path: &Path) -> u64 {
+    let mut header_bytes = [0u8; HEADER_SIZE];
+    File::open(path)
+        .unwrap()
+        .read_exact(&mut header_bytes)
+        .unwrap();
+    HeaderCodec::decode(&header_bytes).unwrap().wal_size
+}
+
 /// Test basic put operation with bytes.
 #[test]
 fn put_bytes_basic() {
@@ -434,3 +446,95 @@ fn timeline_iteration() {
 
     assert_eq!(entries.len(), 3, "Should have 3 timeline entries");
 }
+
+/// Regression test for memvid/memvid#230 — sustained commit-per-put workloads
+/// that span multiple WAL growth cycles must keep the embedded WAL intact.
+///
+/// Before the fix, `grow_wal_region` / `ensure_wal_capacity` updated
+/// `header.footer_offset` and `self.data_end` after shifting the data region
+/// but left the cached `payload_region_end()` value stale. The next call to
+/// `rebuild_indexes` then sought to that pre-growth offset (which now lies
+/// inside the grown WAL region) and overwrote WAL record payloads, producing
+/// `Embedded WAL is corrupted at offset N: wal record checksum mismatch` on
+/// the following commit.
+#[test]
+fn commit_per_put_survives_wal_growth() {
+    let dir = TempDir::new().unwrap();
+    let path = dir.path().join("wal_growth.mv2");
+
+    // Capture the initial WAL size from a freshly created, then closed, store.
+    // The header is read with no writer open so it never races the store's
+    // file lock — on Windows a held byte-range lock makes a second handle's
+    // read fail with "another process has locked a portion of the file".
+    let initial_wal_size = {
+        let mem = Memvid::create(&path).unwrap();
+        drop(mem);
+        read_wal_size(&path)
+    };
+
+    // Use text-indexable payloads of varying length so each commit drives the
+    // full Tantivy rebuild path (`rebuild_indexes` → `flush_tantivy`) that
+    // seeks to `payload_region_end()`. The mix of sizes ensures multiple
+    // WAL growth cycles occur across the run.
+    let words: &[&str] = &[
+        "alpha", "bravo", "charlie", "delta", "echo", "foxtrot", "golf", "hotel", "india",
+        "juliet", "kilo", "lima", "mike", "november", "oscar", "papa", "quebec", "romeo", "sierra",
+        "tango", "uniform", "victor", "whiskey", "x-ray", "yankee", "zulu",
+    ];
+
+    let doc_count = 24u32;
+    let frames_written;
+    {
+        let mut mem = Memvid::open(&path).unwrap();
+        for i in 0..doc_count {
+            // Ramp the body size from ~4 KiB up past the initial 64 KiB WAL
+            // region so the run deterministically crosses several
+            // `grow_wal_region` cycles (rather than relying on incidental
+            // wrap/checkpoint timing). Each commit then runs `rebuild_indexes`,
+            // which seeks to `payload_region_end()` — the path #230 corrupted.
+            let body_len = 4096usize + (i as usize) * 6144;
+            let mut body = String::with_capacity(body_len + 16);
+            let mut idx = i as usize;
+            while body.len() < body_len {
+                body.push_str(words[idx % words.len()]);
+                body.push(' ');
+                idx = idx.wrapping_add(1);
+            }
+            let opts = PutOptions {
+                uri: Some(format!("mv2://wal-growth/doc-{i}")),
+                title: Some(format!("doc-{i}")),
+                search_text: Some(body.clone()),
+                ..Default::default()
+            };
+            mem.put_bytes_with_options(body.as_bytes(), opts)
+                .unwrap_or_else(|e| panic!("put #{i} failed: {e}"));
+            mem.commit()
+                .unwrap_or_else(|e| panic!("commit #{i} failed: {e}"));
+        }
+        // Large `search_text` bodies are chunked, so each doc yields one or
+        // more frames; capture the true total before closing.
+        frames_written = mem.stats().unwrap().frame_count;
+        // mem dropped here, releasing the file lock before we read the header.
+    }
+
+    // The WAL region only ever grows, so reading the size once the writer is
+    // closed reflects the maximum it reached during the run.
+    let final_wal_size = read_wal_size(&path);
+    assert!(
+        final_wal_size > initial_wal_size,
+        "test must exercise WAL growth (initial={initial_wal_size}, final={final_wal_size})"
+    );
+
+    // Reopening forces a full WAL scan; checksum verification will fire here
+    // if any record payload was clobbered by a stale-offset index write.
+    assert!(
+        frames_written >= u64::from(doc_count),
+        "expected at least one frame per doc (got {frames_written} for {doc_count} docs)"
+    );
+    let reopened = Memvid::open_read_only(&path).unwrap();
+    assert_eq!(
+        reopened.stats().unwrap().frame_count,
+        frames_written,
+        "all frames should be durable after WAL growth"
+    );
+}