186 Commits

Author SHA1 Message Date
Usman Akinyemi e6bd9f7b9c Update README.md
CI / Test (macos-latest, stable) (push) Waiting to run
CI / Test (ubuntu-latest, stable) (push) Waiting to run
CI / Test (windows-latest, stable) (push) Waiting to run
CI / Lint (push) Waiting to run
2026-07-14 20:09:06 +05:30
Usman Akinyemi b33aa7a816 Update README.md
Added the correct discord link
2026-07-11 02:54:52 +05:30
Olow304 fa9efe5127 test: make commit_per_put_survives_wal_growth Windows-safe and deterministic
The regression test read the WAL header from a second file handle while the
writer was still open, which fails on Windows ("another process has locked a
portion of the file"). Read the header size only after the writer is dropped,
and ramp payload sizes past the initial 64 KiB WAL region so growth is forced
deterministically instead of relying on wrap/checkpoint timing. Account for
chunked frames when asserting durability.
2026-05-27 15:31:22 -04:00
Olow304 e18fe55494 v2.0.140: fix WAL corruption after region growth (#230) and Tantivy temp-dir leak (#215)
Docker Release / build-and-push (push) Waiting to run
- #230: refresh cached_payload_end after WAL growth so rebuild_indexes no
  longer seeks into the grown WAL region and overwrites record payloads
  (was surfacing as "wal record checksum mismatch").
- #215: make TantivyEngine drop its TempDir last and release the index
  writer first, so per-put working directories are cleaned up instead of
  leaking into the system temp dir (notably on Windows).
- add commit_per_put_survives_wal_growth regression test.
2026-05-27 15:12:48 -04:00
Saleban Olow 178e2772eb clean up - read me 2026-05-06 13:37:48 -05:00
Mr Sharafdin 7460a79766 Merge pull request #206 from nestorfernando3/codex/update-readme-es
docs(i18n): refresh Spanish README sections
2026-03-16 16:23:50 +03:00
Olow304 92b0ec8a9b fix: clippy pedantic — use let-else, allow trivially_copy_pass_by_ref 2026-03-14 11:57:46 -04:00
Nestor Fernando De Leon Llanos 0352770baf docs(i18n): refresh Spanish README sections 2026-03-14 10:51:52 -05:00
Olow304 11a18fdd6b fmt: fix formatting 2026-03-14 11:34:56 -04:00
Olow304 a79dfddc8b v2.0.139: Fix MV005 — graceful handling of stale frame_ids in all search paths
Docker Release / build-and-push (push) Waiting to run
2026-03-13 18:33:13 -04:00
Olow304 e35546ff7a fix: expose lex_enabled/vec_enabled in stats and auto-detect vec on reopen, gracefully skip out-of-range frame ids in timeline (#194, #196)
Docker Release / build-and-push (push) Waiting to run
2026-03-03 16:58:47 -05:00
Olow304 7be69c6a88 Skip xlsx_structured tests when arden.xlsx fixture is absent (CI)
Docker Release / build-and-push (push) Waiting to run
2026-02-15 14:02:18 -05:00
Olow304 223b93d2ee Fix clippy and dead-code lints in xlsx extraction pipeline 2026-02-15 13:51:29 -05:00
Olow304 64bbd32d50 Fix rustfmt formatting in xlsx_chunker, xlsx_ooxml, and xlsx.rs 2026-02-15 13:48:01 -05:00
Olow304 7511889207 Wire up structured XLSX extraction pipeline in XlsxReader 2026-02-15 13:44:14 -05:00
Olow304 97f68b220f Add structured XLSX extraction pipeline with table detection, OOXML metadata parsing, and semantic chunking 2026-02-15 13:31:26 -05:00
Mohamed Mohamed 8d8addb617 Update README.md 2026-02-10 14:58:53 -05:00
Mr Sharafdin d7a2657eef Merge pull request #193 from nightire/translate/readme-zh-cn
(docs): translate README.md into Simplified Chinese #104
2026-02-07 18:31:29 +03:00
Albert Yu a40dbf3843 (docs): translate README.md into Simplified Chinese #104 2026-02-07 22:56:08 +08:00
Olow304 f80753463c fix: resolve clippy pedantic lints and add missing VecIndexManifest model field 2026-02-06 19:04:41 -05:00
Olow304 c84da0bb72 fix: clippy linting error 2026-02-06 18:56:18 -05:00
Olow304 4e04ed8282 Add frame-level ACL and enforcement plumbing across search/ask/replay with robustness fixes, tests, and benchmark/example updates.
Docker Release / build-and-push (push) Waiting to run
2026-02-06 18:42:01 -05:00
Mohamed Mohamed 38cdd32bb2 Update README.md 2026-02-04 14:49:55 -05:00
sharafdin 724be9f054 Move deprecation warning to bottom of README 2026-02-04 22:39:16 +03:00
sharafdin d7bd970982 docs: add Memvid v1 deprecation warning to README
QR-based memory (v1) is deprecated. Added prominent warning
linking to https://docs.memvid.com/memvid-v1-deprecation
2026-02-04 16:50:26 +03:00
Mohamed Mohamed 3864ee8332 Update README.md 2026-02-03 15:59:46 -05:00
Mohamed Mohamed 67cedb353e Update README.md 2026-02-03 15:56:51 -05:00
Mohamed Mohamed f8e661e88c Update README.md 2026-02-03 15:55:21 -05:00
Mohamed Mohamed ae528a4650 Update README.md 2026-02-03 15:45:55 -05:00
Mohamed Mohamed e2aeee58b3 Update README.md 2026-02-03 15:33:33 -05:00
Mohamed Mohamed 9ecafb9f1e Update README.md 2026-02-03 15:28:49 -05:00
Saleban Olow df8723a14e Merge pull request #188 from 0x-pankaj/feat/vector-model-consistency
feat: enforce vector index model consistency
2026-01-29 10:47:13 -05:00
pankaj 80a254459b feat: enforce vector index model consistency via strict binding 2026-01-28 04:12:18 +05:45
Olow304 582556db83 Fix: run cargo fmt on clip.rs and text_embed.rs
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-27 11:12:30 -05:00
Olow304 5e5673b36d Fix CI: move target-specific deps section after main dependencies
The [target.'cfg(target_os = "macos")'.dependencies] section was placed
in the middle of the main dependencies, causing all subsequent deps
(tantivy, wide, etc.) to be parsed as macOS-only. This made the lex
and simd features fail on Linux/Windows.

Move the target-specific section to after all main dependencies.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-27 10:48:14 -05:00
Olow304 cbe2f3efdd Commit Cargo.lock for reproducible CI builds
Cargo's dependency resolution was not including tantivy and wide
on Ubuntu CI despite the lex and simd features being enabled.
This appears to be a Cargo bug or edge case with optional dependencies.

Committing Cargo.lock ensures consistent dependency resolution
across all CI runners.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-27 10:39:06 -05:00
Olow304 9b7f5a62a1 Fix CI: use Cargo.toml hash for cache key instead of missing Cargo.lock
The previous cache key used hashFiles('**/Cargo.lock') but Cargo.lock is
gitignored in this library repo. This caused the cache to have an empty
hash suffix, restoring stale dependencies that didn't include tantivy,
wide, and other optional deps that were added recently.

Using Cargo.toml hash ensures the cache invalidates when dependencies change.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-27 10:33:37 -05:00
Olow304 a4fc2b4e72 Fix CI build: add missing #[cfg(feature = lex)] guards for tantivy-dependent code 2026-01-27 09:44:02 -05:00
Olow304 87c1343f4d Fix: suppress ONNX Runtime stderr warnings on macOS 2026-01-27 09:03:19 -05:00
Saleban Olow f24b90db5d Merge pull request #187 from 0x-pankaj/feat/fix-symspell-cleanup 2026-01-26 20:48:13 -05:00
pankaj aaae5a02ee feat: fix symspell_cleanup data corruption and add dictionary download tooling 2026-01-26 22:08:45 +05:45
Olow304 dbca79e89d fix: resolve redundant closure lint in tantivy.rs:159 2026-01-25 16:05:10 -05:00
Olow304 9108395e48 fix: resolve redundant closure lint in search/mod.rs 2026-01-25 15:54:22 -05:00
Olow304 2bbad84152 fix: Lint formatting issues fixed. 2026-01-25 14:34:01 -05:00
Olow304 06159b6629 Release: v2.0.135 memvid
Docker Release / build-and-push (push) Waiting to run
2026-01-25 14:22:56 -05:00
Saleban Olow bb8eb611f6 Merge pull request #186 from 0x-pankaj/fix/windows-tantivy-file-locking
fix(tests): add Windows delay for Tantivy file handle release
2026-01-25 13:54:20 -05:00
pankaj f5df80e165 fix(tests): apply Windows Tantivy release delay to doctor recovery tests 2026-01-26 00:06:05 +05:45
pankaj bec7de89d9 fix(tests): add Windows delay for Tantivy file handle release 2026-01-25 23:32:02 +05:45
Mr Sharafdin c26911bead Merge pull request #185 from 0x-pankaj/feat/hnsw-implementation
feat: apply HNSW implementation patch
2026-01-24 18:28:31 -08:00
Mr Sharafdin 5bf7d92f6d Merge pull request #173 from 0x-pankaj/feat/api-embedding-providers
feat: add OpenAI API embedding provider
2026-01-24 18:17:44 -08:00
Mr Sharafdin 0bb56a0b01 Merge pull request #175 from 0x-pankaj/fix/extraction-cache-lru
feat(extract): add LRU eviction to extraction cache
2026-01-24 18:12:35 -08:00
pankaj 1cbfcacf6e feat: implement HNSW vector search with robust fixed-point metric 2026-01-25 07:48:12 +05:45
pankaj 7a2e55f57e Merge remote-tracking branch 'origin/main' into fix/extraction-cache-lru 2026-01-25 05:24:59 +05:45
pankaj 897ed7c0ac Merge branch 'main' into feat/api-embedding-providers
# Conflicts:
#	Cargo.toml
2026-01-25 04:51:07 +05:45
Mr Sharafdin 24a31ee3a1 Merge pull request #184 from Adam-Elmi/main
Created a script to add flags
2026-01-24 13:19:14 -08:00
Adam-Elmi a2a13438ab Created a script to add flags and the LANG_MAP in the script includes 183 languages from the ISO 639-1 standard: https://en.wikipedia.org/wiki/List_of_ISO_639_language_codes 2026-01-24 21:14:11 +03:00
Mr Sharafdin 950e51ffca Merge pull request #181 from Adam-Elmi/main
Added HTML markers to all README files to make updates easier.
2026-01-24 09:11:33 -08:00
Mr Sharafdin 04fe0968e3 Merge pull request #182 from krishnaK-D-Bair/readme-bn-translation
docs: add Bengali (bn) README translation
2026-01-24 08:54:56 -08:00
Saleban Olow 1480ea901a Merge pull request #180 from 0x-pankaj/fix/clippy-safety-overhaul
Fix/clippy safety overhaul
2026-01-24 11:35:26 -05:00
pankaj 9a7b3c2275 fix: resolve all clippy lints and verify tests passing 2026-01-24 21:26:31 +05:45
Krishna Bairagya f328e0046d docs: add Bengali (bn) README translation 2026-01-24 20:10:54 +05:30
Adam-Elmi 57745a71c9 Updated Somali readme with markers and links 2026-01-24 17:00:09 +03:00
Adam-Elmi 3f528bd6b2 Updated Dutch readme with markers and links 2026-01-24 17:00:09 +03:00
Adam-Elmi 4292c1de8f Updated Korean readme with markers and links 2026-01-24 17:00:09 +03:00
Adam-Elmi d328891196 Updated Japanese readme with markers and links 2026-01-24 17:00:09 +03:00
Adam-Elmi f55e0dd613 Updated Hindi readme with markers and links 2026-01-24 17:00:09 +03:00
Adam-Elmi 367660c0fb Updated French readme with markers and links 2026-01-24 17:00:08 +03:00
Adam-Elmi 9aa39dc092 Updated Spanish readme with markers and links 2026-01-24 17:00:08 +03:00
Adam-Elmi 169baf4da6 Updated Czech readme with markers and links 2026-01-24 17:00:08 +03:00
Adam-Elmi 4a0363e1bb Updated Arabic readme with markers and links 2026-01-24 17:00:08 +03:00
Adam-Elmi 10b78c5a87 Created script to update translations 2026-01-24 17:00:08 +03:00
Adam-Elmi e143cb74b6 Added header markers and english link to main readme 2026-01-24 17:00:08 +03:00
pankaj 3461877851 fix(clippy): surgical fixes with documented allows & cast safety 2026-01-24 19:01:31 +05:45
pankaj 50bff53f4a fix(clippy): resolve all unwrap/expect errors with module-level allows for verified-safe patterns 2026-01-24 18:58:57 +05:45
pankaj 53d8d95a06 fix(clippy): resolve Tier 1 unwrap/expect errors with safe byte extraction helpers 2026-01-24 18:58:57 +05:45
Mr Sharafdin 37df42fc08 Merge pull request #169 from 0x-pankaj/feat/whisper-quantization
feat: add Whisper model quantization support and fix rubato 1.0 compa…
2026-01-24 03:09:12 -08:00
Saleban Olow ee1bebd623 Merge pull request #178 from Abhisheklearn12/fix/implicit-and-default
Fix: Change implicit query operator from OR to AND for precision
2026-01-23 09:18:20 -05:00
Abhisheklearn12 f129a948a5 search(parser): use implicit AND for multi-word queries to improve precision 2026-01-23 13:27:09 +05:30
Saleban Olow 428f1c41e1 Merge pull request #176 from 0x-pankaj/feature/simd-distance-acceleration
feat: add SIMD acceleration for vector distance calculations
2026-01-21 16:22:50 -05:00
Mr Sharafdin 064445129e Merge pull request #177 from yukaty/add-japanese-readme
docs(i18n): add Japanese README translation
2026-01-21 12:47:47 -08:00
yukaty b047f44d07 docs(i18n): add Japanese README translation 2026-01-21 12:39:06 -07:00
pankaj 452de831b3 feat: add SIMD acceleration for vector distance calculations 2026-01-20 15:35:37 +05:45
pankaj 6801712d73 feat(extract): add LRU eviction to extraction cache 2026-01-20 11:27:32 +05:45
Mr Sharafdin 7851bf253f Merge pull request #167 from orbisai0security/fix-v-002-src-replay-types.rs
[Security] Enhance Input/Output Handling with Size Limits and Sanitization for Robustness Against DoS
2026-01-18 02:49:01 -08:00
pankaj 12e9c2a954 feat: add OpenAI API embedding provider 2026-01-17 23:11:14 +05:45
Mr Sharafdin bc613be265 Merge pull request #170 from akash-R-A-J/test/doctor-edge-cases
test(doctor): add edge case and invariant coverage
2026-01-17 05:10:43 -08:00
Mr Sharafdin bb8224f23a Disable Dependabot configuration file 2026-01-17 15:58:24 +03:00
Akash Raj 62299297a1 test(doctor): add vacuum ordering and footer invariant tests 2026-01-17 02:59:38 +05:30
Akash Raj 3ceb707ea2 test(doctor): add dry_run and bounds checking tests 2026-01-17 02:42:25 +05:30
Olow304 8b9cd22e00 fix: default multi-word queries to OR for better recall and use search_text in parallel segment indexing, encryption capsule
Docker Release / build-and-push (push) Waiting to run
2026-01-16 12:04:07 -05:00
pankaj a3c1be7bbe feat: add Whisper model quantization support and fix rubato 1.0 compatibility 2026-01-16 22:22:16 +05:45
orbisai0security a45feedd18 fix: resolve critical vulnerability V-002
Automatically generated security fix
2026-01-16 03:54:49 +00:00
sharafdin 8ad41268c9 added install script 2026-01-16 04:47:36 +03:00
Mr Sharafdin 5b4ecba6e5 Merge pull request #165 from Sjondepon/main
Dutch Readme added
2026-01-15 17:39:18 -08:00
Saleban Olow fb06576fbb Merge pull request #163 from akash-R-A-J/fix/stabilize-ci-tests 2026-01-15 20:38:06 -05:00
Sjondepon f1c5df2fe7 Merge branch 'main' into main 2026-01-15 22:59:57 +01:00
Sjondepon 007d954993 Update README.md 2026-01-15 22:54:55 +01:00
Sjondepon 5ff63270f1 Update README.nl.md 2026-01-15 22:52:24 +01:00
Sjondepon aa4748d6f2 Update README.nl.md 2026-01-15 22:44:06 +01:00
Sjondepon 09dabe3853 Update README.nl.md 2026-01-15 22:35:44 +01:00
Akash Raj 17f4b91ae1 fix: stabilize CI for Ubuntu, macOS, and Windows 2026-01-16 00:09:42 +05:30
Sjondepon 718bfbb573 Update README.nl.md 2026-01-15 16:14:03 +01:00
Sjondepon 3db67879a4 Update README.nl.md 2026-01-15 14:12:12 +01:00
Mr Sharafdin ec35216c73 Merge pull request #160 from qool/docs/i18n-cs
docs(i18n): add Czech README translation
2026-01-15 04:49:04 -08:00
Mikuláš Sychra 22e0e6d100 docs(i18n): add Czech README translation 2026-01-15 12:40:45 +01:00
sharafdin e7bd81cb93 fix: correct dependabot cooldown configuration. 2026-01-15 14:40:23 +03:00
Mr Sharafdin 6f5b07b5ba Merge pull request #153 from reneleonhardt/chore/update-dependencies
chore: update dependencies
2026-01-15 03:35:07 -08:00
Mr Sharafdin 28fa10dc4b Merge pull request #158 from fa-ina-tic/main
docs(i18n): add Korean README translation
2026-01-15 02:58:04 -08:00
soomin.lee 2915f2ed7c docs(i18n): add Korean README translation
docs(i18n): add Korean README translation
2026-01-15 16:23:25 +09:00
Saleban Olow 0d4692043e Merge pull request #150 from 0x-pankaj/feat/return-metadata-in-search-results 2026-01-14 21:04:24 -05:00
Saleban Olow 1cae3ea29a Merge pull request #157 from 0x-pankaj/feat/embedding-cache 2026-01-14 21:03:05 -05:00
pankaj e759a5a7ec feat: add embedding cache with LRU eviction for instant retrieval of repeated texts 2026-01-15 03:41:10 +05:45
Saleban Olow 6b924470ff Merge pull request #156 from akash-R-A-J/fix/text-embedding-feature-gate
fix: gate text_embedding example behind vec feature
2026-01-14 16:50:51 -05:00
Saleban Olow c2ea9bd80e Merge pull request #154 from akash-R-A-J/test/doctor-recovery-tests
test(doctor): add corruption recovery path tests
2026-01-14 16:50:24 -05:00
Akash Raj 1d9aa25b73 fix: gate text_embedding example behind vec featur
e
2026-01-15 02:23:12 +05:30
Akash Raj c7025beb44 fix: gate text_embedding example behind vec feature 2026-01-15 01:55:32 +05:30
Akash Raj a690287713 test(doctor): add corruption recovery path tests 2026-01-15 00:17:35 +05:30
Rene Leonhardt 68814d8a8c chore: update dependencies 2026-01-14 17:40:06 +01:00
Mr Sharafdin fc77af0c07 Merge pull request #151 from 0x-pankaj/feat/local-text-embeddings
feat: add local text embedding provider with ONNX support
2026-01-14 07:51:21 -08:00
Mohamed Mohamed cea051ce65 Update README.md 2026-01-13 18:30:00 -05:00
Sjondepon 6cd842433c Update README.nl.md 2026-01-13 23:23:25 +01:00
Sjondepon d64084c8ba Update README.nl.md 2026-01-13 23:11:30 +01:00
Sjondepon 9cc3b9709d Update README.nl.md 2026-01-13 23:09:39 +01:00
Sjondepon ca3f039b9e Update README.nl.md 2026-01-13 23:07:38 +01:00
Sjondepon be782f9378 Create README.nl.md 2026-01-13 23:06:20 +01:00
pankaj 3b8e7494d0 feat: add local text embedding provider with ONNX support 2026-01-14 02:24:00 +05:45
Saleban Olow fa5b8287bf Merge pull request #149 from akash-R-A-J/feat/encryption-edge-tests
test(encryption): add edge case tests
2026-01-12 18:08:55 -05:00
pankaj afbf8d76da feat: return extra_metadata in search results
Closes #147

- Add extra_metadata: BTreeMap<String, String> field to SearchHitMetadata
- Populate extra_metadata from Frame.extra_metadata in all search handlers
- Allows users to retrieve custom metadata stored via PutOptions.extra_metadata

Files changed:
- src/types/search.rs: New field in SearchHitMetadata
- src/memvid/search/fallback.rs: Populate in 2 initializers
- src/memvid/search/api.rs: Populate in vector search
- src/memvid/search/tantivy.rs: Populate in Tantivy search
2026-01-13 04:39:09 +05:45
Akash Raj db09594314 test(encryption): add edge case tests 2026-01-13 04:12:27 +05:30
Saleban Olow 817c52f315 Merge pull request #146 from akash-R-A-J/feat/encryption-error-tests
test(encryption): add error handling tests
2026-01-12 15:51:41 -05:00
Akash Raj c3bb56e27c test(encryption): add error handling tests 2026-01-12 23:56:46 +05:30
Mr Sharafdin f8cbc9c7df Merge pull request #139 from akash-R-A-J/feat/encryption-tests
test(encryption): add backward compatibility tests for legacy format
2026-01-12 02:05:43 -08:00
Mr Sharafdin 5deb894626 Merge pull request #141 from memvid/dockerize
feat(docker): add core library dockerization
2026-01-12 01:44:56 -08:00
sharafdin 5c1a984f6d feat(docker): add core library dockerization
- Add Dockerfile for production builds
- Add Dockerfile.dev for development
- Add Dockerfile.test for testing
- Add docker-compose.yml for easy workflow
- Organize in docker/core/ directory
2026-01-12 12:43:53 +03:00
Akash Raj 928f55d7f4 style: format doctor.rs with rustfmt 2026-01-12 06:30:15 +05:30
Akash Raj 158dc3a425 test(encryption): implement backward compatibility tests 2026-01-12 04:31:46 +05:30
Akash Raj 96ad653479 test(encryption): add legacy format tests and fixtures 2026-01-11 23:08:43 +05:30
Akash Raj 7a885391af test(encryption): add skeleton tests for streaming encryption 2026-01-11 05:10:32 +05:30
Olow304 b7cb53f46f feat(doctor): add quiet mode to suppress debug logs and add streaming encryption tests pr tested
Docker Release / build-and-push (push) Waiting to run
2026-01-10 17:10:02 -05:00
Mr Sharafdin eb4a3afd5d Merge pull request #117 from akash-R-A-J/main
fix(encryption): add streaming encryption to prevent OOM on large files
2026-01-10 13:31:11 -08:00
Mr Sharafdin b01b9a9621 Merge pull request #136 from krishnaK-D-Bair/feature-hindi-readme
Add Hindi translation for README (hi) (Fixes #112)
2026-01-10 03:01:47 -08:00
Krishna Bairagya 3d80b983a4 Fix minor issues in Hindi README 2026-01-10 12:34:41 +05:30
Krishna Bairagya b08a5e16f1 Add Hindi translation for README (hi) (Fixes #112) 2026-01-10 12:15:12 +05:30
Mohamed Mohamed 4492586088 Update README.md 2026-01-09 19:54:26 -05:00
Mr Sharafdin f363e4fdf6 Merge pull request #135 from omartood/main
docs(i18n): add Arabic README translation
2026-01-09 15:09:29 -08:00
Olow304 5dfcab903b fix(ci): allow non_std_lazy_statics lint for now
Docker Release / build-and-push (push) Waiting to run
2026-01-09 16:37:14 -05:00
Olow304 36d6bd6cd2 fix(ci): install rustfmt explicitly, skip crash_recovery on Windows 2026-01-09 16:33:40 -05:00
Olow304 b50fc74a78 fix(ci): skip Windows-incompatible tests, remove redundant toolchain 2026-01-09 16:20:31 -05:00
omartood db34b10572 docs(i18n): add Arabic README translation 2026-01-09 23:33:52 +03:00
Mr Sharafdin 3ef94bb1fb Merge pull request #134 from GinoGreen/docs/i18n-it
Add Italian README translation
2026-01-09 09:11:00 -08:00
Olow304 44b2aa21de feat: add Ed25519 ticket signature verification for dashboard-issued capacity tickets 2026-01-09 12:03:48 -05:00
GinoGreen d51a61f31a docs(i18n): add Italian language selector README 2026-01-09 17:54:00 +01:00
GinoGreen 3d2f5cd5d0 docs(i18n): add it README translation 2026-01-09 17:48:58 +01:00
Akash Raj 8b719759dc Merge branch 'memvid:main' into main 2026-01-09 21:40:13 +05:30
Mr Sharafdin 715c4271bc Merge pull request #119 from DavidReque/feature/README-translation-spanish
Add Spanish README translation
2026-01-08 23:14:41 -08:00
David Requeno 007d975427 docs(i18n): add Spanish README translation 2026-01-08 21:33:23 -06:00
Mr Sharafdin 2fa594ff1b Merge pull request #118 from shiinedev/docs/i18n-Somali-so
docs(i18n): add Somali README link
2026-01-08 13:19:30 -08:00
shiinedev 6de17114e7 docs(i18n): add Soomaali README link 2026-01-08 21:12:17 +00:00
Mr Sharafdin 1b1a0ba48e Merge pull request #116 from shiinedev/docs/i18n-Somali-so
docs(i18n): add <Somali> README translation
2026-01-08 13:04:04 -08:00
shiinedev 13dab15c4f fix:some issue for somali translation README 2026-01-08 20:42:08 +00:00
Akash Raj cc1d3bcd10 Style(encryption): apply rustfmt 2026-01-09 02:07:36 +05:30
shiinedev c807120af4 docs(i18n): add <Somali> README translation 2026-01-08 20:17:36 +00:00
Akash Raj 6d924a6d68 Refactor(encryption): make write_atomic generic to preserve decryption errors 2026-01-09 01:11:44 +05:30
Akash Raj bfd0fbca55 Refactor(encryption): add backward-compatible auto-detection for streaming format 2026-01-08 23:42:34 +05:30
Akash Raj 68bcbb382f Expose streaming mv2e encryption APIs alongside existing capsule interface 2026-01-08 22:07:04 +05:30
Akash Raj e099b7c1d1 Fix OOM issue by streaming file encryption and decryption 2026-01-08 22:02:20 +05:30
Mr Sharafdin 4aef373024 docs(i18n): add Français README link 2026-01-08 18:48:56 +03:00
Mr Sharafdin 436f9e7213 Replace old Trendshift badge with new link (1 Repo of the Day)
Updated Trendshift badge link in README.
2026-01-08 17:19:43 +03:00
Mr Sharafdin a8ff3e27ff Merge pull request #113 from karamokoisrael/docs/i18n/french-translation
Add French README translation
2026-01-08 00:16:08 -08:00
Karamoko Israel 72eef1e5cf docs(i18n): add French README translation 2026-01-08 15:24:37 +08:00
sharafdin 787eee177e Add translation contribution links and format CONTRIBUTING_TRANSLATIONS.md 2026-01-08 08:13:35 +03:00
Mr Sharafdin 79f99d822a Add internationalization README for translations
This README provides guidelines for translating the main README into various languages, including structure, contributing instructions, and language codes.
2026-01-08 06:57:40 +03:00
Mr Sharafdin 8c5a964e26 Update README.md 2026-01-07 22:51:16 +03:00
Mr Sharafdin 45ea5c6f0e Add Trendshift badge to README
Added Trendshift badge to README for repository tracking.
2026-01-07 22:50:13 +03:00
sharafdin 55630f2ede Update Makefile: add all 4 examples with path arguments 2026-01-07 18:22:30 +03:00
sharafdin 4e99b40fdb Update examples to accept file paths as arguments 2026-01-07 18:20:02 +03:00
Saleban Olow 6044d806b4 Remove CI badge from README.md
Removed CI badge link from README
2026-01-05 15:00:46 -06:00
Mr Sharafdin b792821293 Change copyright year to 2026
Updated copyright year in LICENSE file.
2026-01-05 21:02:57 +03:00
Olow304 e2588511ac fix: use correct rust-toolchain action name
dtolnay/rust-action doesn't exist, the correct action is dtolnay/rust-toolchain

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-05 10:33:52 -05:00
Mr Sharafdin 1ae228a642 Fix Discord link in README
Updated Discord link in README.md
2026-01-05 15:26:53 +03:00
Mr Sharafdin 358734e78c Update Discord badge link in README.md 2026-01-05 12:53:47 +03:00
sharafdin 1aff144772 Add Makefile with common development tasks
- Build commands (debug, release, all features)
- Test commands (unit, integration, verbose)
- Code quality (fmt, clippy, lint)
- Example runners for all examples
- Docker build and test commands
- CI pipeline target
- Documentation generation
- Utility commands (clean, update, audit, version)
2026-01-05 12:34:30 +03:00
sharafdin ad0b1a7db5 Add GitHub Actions workflow for Docker releases 2026-01-05 12:30:03 +03:00
sharafdin 047067e34d Add Docker test script 2026-01-05 12:29:03 +03:00
sharafdin e18e818885 Add Docker documentation and usage guides 2026-01-05 12:28:50 +03:00
sharafdin 1a750614c6 Add Dockerfile and .dockerignore for Memvid CLI 2026-01-05 12:28:22 +03:00
180 changed files with 351183 additions and 1858 deletions
+42
View File
@@ -0,0 +1,42 @@
# Git
.git
.gitignore
.github
# Rust
target/
# Documentation
*.md
docs/
CHANGELOG.md
SECURITY.md
CODE_OF_CONDUCT.md
CONTRIBUTING.md
# IDE
.vscode/
.idea/
*.swp
*.swo
*~
# OS
.DS_Store
Thumbs.db
# Docker
docker/
Dockerfile*
docker-compose*.yml
.dockerignore
# Test artifacts
*.log
*.tmp
# Build artifacts
*.o
*.so
*.dylib
*.dll
+31
View File
@@ -0,0 +1,31 @@
version: 2
updates:
- package-ecosystem: cargo
directory: /
schedule:
interval: weekly
cooldown: # applies only to version-updates (not security-updates)
default-days: 7
ignore:
- dependency-name: "bincode"
versions: [">=3.0.0"] # unsupported since v3
- package-ecosystem: docker
directories:
- docker/cli
- docker/core
schedule:
interval: weekly
cooldown:
default-days: 7
- package-ecosystem: docker-compose
directory: docker/core
schedule:
interval: weekly
cooldown:
default-days: 7
- package-ecosystem: github-actions
directory: /
schedule:
interval: weekly
cooldown: # https://blog.yossarian.net/2025/11/21/We-should-all-be-using-dependency-cooldowns
default-days: 7
+11 -11
View File
@@ -20,15 +20,15 @@ jobs:
rust: [stable]
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@8e8c483db84b4bee98b60c0593521ed34d9990e8 # v6.0.1
- name: Install Rust
uses: dtolnay/rust-action@stable
uses: dtolnay/rust-toolchain@stable
with:
toolchain: ${{ matrix.rust }}
- name: Cache cargo
uses: actions/cache@v4
uses: actions/cache@9255dc7a253b0ccc959486e2bca901246202afeb # v5.0.1
with:
path: |
~/.cargo/bin/
@@ -36,28 +36,28 @@ jobs:
~/.cargo/registry/cache/
~/.cargo/git/db/
target/
key: ${{ runner.os }}-cargo-${{ hashFiles('**/Cargo.lock') }}
key: ${{ runner.os }}-cargo-${{ hashFiles('**/Cargo.toml') }}
- name: Build
run: cargo build --verbose
- name: Run tests
run: cargo test --verbose
run: cargo test --verbose ${{ runner.os == 'Windows' && '-- --test-threads=1' || '' }}
lint:
name: Lint
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@8e8c483db84b4bee98b60c0593521ed34d9990e8 # v6.0.1
- name: Install Rust
uses: dtolnay/rust-action@stable
with:
toolchain: stable
components: rustfmt, clippy
uses: dtolnay/rust-toolchain@stable
- name: Install components
run: rustup component add rustfmt clippy
- name: Check formatting
run: cargo fmt --all -- --check
- name: Clippy
run: cargo clippy -- -D warnings
run: cargo clippy -- -D warnings -A clippy::non_std_lazy_statics
+50
View File
@@ -0,0 +1,50 @@
name: Docker Release
on:
push:
tags:
- 'v*'
workflow_dispatch:
env:
REGISTRY: docker.io
IMAGE_NAME: memvid/cli
jobs:
build-and-push:
runs-on: ubuntu-latest
permissions:
contents: read
packages: write
steps:
- name: Checkout
uses: actions/checkout@8e8c483db84b4bee98b60c0593521ed34d9990e8 # v6.0.1
- name: Set up QEMU
uses: docker/setup-qemu-action@c7c53464625b32c7a7e944ae62b3e17d2b600130 # v3.7.0
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@8d2750c68a42422c14e847fe6c8ac0403b4cbd6f # v3.12.0
- name: Login to Docker Hub
uses: docker/login-action@5e57cd118135c172c3672efd75eb46360885c0ef # v3.6.0
with:
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}
- name: Extract version
id: version
run: echo "VERSION=${GITHUB_REF#refs/tags/v}" >> $GITHUB_OUTPUT
- name: Build and push
uses: docker/build-push-action@263435318d21b8e681c14492fe198d362a7d2c83 # v6.18.0
with:
context: ./docker/cli
platforms: linux/amd64,linux/arm64
push: true
tags: |
${{ env.IMAGE_NAME }}:latest
${{ env.IMAGE_NAME }}:${{ steps.version.outputs.VERSION }}
cache-from: type=gha
cache-to: type=gha,mode=max
+8 -2
View File
@@ -17,7 +17,7 @@ doc.mv2
/target
**/*.rs.bk
Cargo.lock
# Cargo.lock - now committed for reproducible CI builds
.idea
@@ -29,7 +29,7 @@ vector_embedding.txt
*.pdf
*.txt
# Exception: SymSpell dictionary files needed for PDF text cleanup
!crates/memvid-core/data/*.txt
!data/*.txt
*.docx
*.doc
*.pptx
@@ -41,3 +41,9 @@ vector_embedding.txt
models/
*.DS_Store
issue_overview.md
feature_100x_memvid.md
plan_model_robust.md
+14 -1
View File
@@ -76,10 +76,19 @@ git commit -m "docs: update README examples"
- Follow standard Rust idioms and conventions
- Use `rustfmt` for formatting (`cargo fmt`)
- Use `clippy` for linting (`cargo clippy`)
- Use `clippy` for linting (`cargo clippy`). We maintain a **zero-warning policy**.
- Prefer explicit types for public APIs
- Use `thiserror` for error definitions
### Linting & Safety
We enforce strict linting to ensure safety and portability:
1. **Zero Warnings**: CI will fail on any warning. Run `cargo clippy --workspace --all-targets -- -D warnings` locally.
2. **No Panics**: `unwrap()` and `expect()` are **denied** in library code. Use `Result` propagation (`?`) or graceful error handling. They are allowed in `tests/`.
3. **No Truncation**: `cast_possible_truncation` is denied. Use `try_from` when converting `u64` to `usize`/`u32`.
4. **Exceptions**: We allow pragmatic lints (e.g., `cast_precision_loss` for ML math) in `src/lib.rs`. Do not add global `#![allow]` without discussion.
### Documentation
- Add doc comments (`///`) to all public functions, structs, and modules
@@ -129,6 +138,10 @@ When reporting bugs, please include:
- Minimal code to reproduce
- Expected vs actual behavior
## Translations
Interested in translating Memvid's documentation? See [Contributing Translations](docs/i18n/CONTRIBUTING_TRANSLATIONS.md) for guidelines on translating the README and other documentation.
## Getting Help
- Open a [Discussion](https://github.com/memvid/memvid/discussions) for questions
Generated
+6837
View File
File diff suppressed because it is too large Load Diff
+60 -18
View File
@@ -1,6 +1,6 @@
[package]
name = "memvid-core"
version = "2.0.131"
version = "2.0.140"
edition = "2024"
rust-version = "1.85.0"
license = "Apache-2.0"
@@ -19,14 +19,14 @@ bincode = { version = "2.0.1", features = ["serde"] }
blake3 = "1.5.1"
uuid = { version = "1.10.0", features = ["v4", "serde"] }
log = "0.4.22"
thiserror = "1.0.63"
thiserror = "2.0.17"
fs2 = "0.4.3"
zstd = "0.13.1"
lz4_flex = "0.11.5"
lz4_flex = "0.12.0"
tracing = "0.1.41"
serde_json = "1.0.145"
ed25519-dalek = { version = "2.2.0", features = ["std"] }
base64 = "0.21.7"
base64 = "0.22.1"
sha2 = "0.10.9"
hex = "0.4.3"
# extractous uses GraalVM native compilation which doesn't work on Windows ARM or WSL2 on ARM
@@ -36,14 +36,14 @@ regex = "1.11.1"
time = { version = "0.3.36", features = ["formatting", "parsing"] }
chrono = { version = "0.4.42", features = ["serde"] }
interim = { version = "0.2.1", optional = true }
lopdf = "0.38"
lopdf = "0.39"
# Pure Rust PDF text extraction - used as primary extractor when extractous is disabled
pdf-extract = { version = "0.10", optional = true }
# High-accuracy PDF text extraction with perfect word spacing (2025)
pdf_oxide = { version = "0.2", optional = true }
pdf_oxide = { version = "0.3", optional = true }
unicode-normalization = "0.1"
unicode-segmentation = "1.11"
zip = { version = "0.6", default-features = false, features = ["deflate"] }
zip = { version = "7.1", default-features = false, features = ["deflate"] }
quick-xml = "0.31"
calamine = "0.22"
pdfium-render = { version = "0.8.28", optional = true }
@@ -53,18 +53,18 @@ crossbeam-channel = { version = "0.5.13", optional = true }
memmap2 = "0.9"
memchr = "2.7"
same-file = "1.0"
fs-err = "2.11"
atomic-write-file = "0.2"
fs-err = "3.2"
atomic-write-file = "0.3"
dirs-next = "2.0"
tantivy = { version = "0.24.2", optional = true, default-features = false, features = ["mmap"] }
ort = { version = "2.0.0-rc.10", optional = true }
hnsw = { version = "0.11.0", optional = true }
smallvec = { version = "1.13", features = ["serde", "union", "const_generics", "write"] }
tantivy = { version = "0.25.0", optional = true, default-features = false, features = ["mmap"] }
ort = { version = "=2.0.0-rc.10", optional = true }
hnsw = { version = "0.11.0", optional = true, features = ["serde"] }
jsonwebtoken = { version = "10.0.0", optional = true, features = ["rust_crypto"] }
image = { version = "0.25", optional = true, default-features = false, features = ["jpeg", "png", "webp"] }
ndarray = { version = "0.16", optional = true }
rayon = { version = "1.10", optional = true }
tokenizers = { version = "0.20", optional = true }
tokenizers = { version = "0.22", optional = true }
symphonia = { version = "0.5.3", optional = true, default-features = false, features = ["aac", "mp3", "flac", "isomp4", "ogg", "wav", "pcm"] }
rubato = { version = "0.15", optional = true }
rustfft = { version = "6.2", optional = true }
@@ -72,7 +72,8 @@ rustfft = { version = "6.2", optional = true }
# Encryption capsules (.mv2e) - feature-gated
argon2 = { version = "0.5", optional = true }
aes-gcm = { version = "0.10", optional = true }
rand = { version = "0.8", optional = true }
rand = { version = "0.8", optional = true, features = ["serde1"] }
rand_pcg = { version = "0.3", optional = true, features = ["serde1"] }
zeroize = { version = "1.7", optional = true }
# Candle ML framework for Whisper transcription
@@ -84,18 +85,28 @@ byteorder = { version = "1.5", optional = true }
# SymSpell for PDF text cleanup - fixes broken word spacing from PDF extraction
symspell = { version = "0.4", optional = true }
# SIMD acceleration for vector distance calculations
wide = { version = "1.1", optional = true }
space = { version = "0.17", optional = true }
# HTTP client for API-based embedding providers (OpenAI, etc.)
reqwest = { version = "0.12", optional = true, default-features = false, features = ["blocking", "json", "rustls-tls"] }
# Platform-specific: libc for stderr suppression on macOS
[target.'cfg(target_os = "macos")'.dependencies]
libc = "0.2"
[features]
default = ["lex", "pdf_extract"]
default = ["lex", "pdf_extract", "simd"]
# pdf_oxide disabled - cff-parser panics on ligature fonts (uniFB01/uniFB02)
# symspell_cleanup disabled - needs more work on preserving numbers/proper nouns
# symspell_cleanup - enables robust PDF text repair (requires `make download-models` for dictionaries)
lex = ["dep:tantivy"]
extractous = ["dep:extractous"]
# Pure Rust PDF extraction - faster than lopdf for text extraction, cross-platform
pdf_extract = ["dep:pdf-extract"]
# High-accuracy PDF extraction with perfect word spacing (recommended for 2025+)
pdf_oxide = ["dep:pdf_oxide"]
vec = ["dep:ort", "dep:hnsw"]
vec = ["dep:ort", "dep:hnsw", "dep:ndarray", "dep:tokenizers", "dep:space", "dep:rand", "dep:rand_pcg"]
clip = ["vec", "dep:image", "dep:ndarray", "dep:rayon", "dep:tokenizers"]
mmap = []
pdfium = ["dep:pdfium-render"]
@@ -116,7 +127,38 @@ replay = []
encryption = ["dep:argon2", "dep:aes-gcm", "dep:rand", "dep:zeroize"]
# SymSpell-based PDF text cleanup - fixes broken word spacing
symspell_cleanup = ["dep:symspell"]
# API-based embedding providers (OpenAI, Anthropic, etc.) - requires network
api_embed = ["dep:reqwest"]
# SIMD acceleration for vector distance calculations
simd = ["dep:wide"]
hnsw_bench = ["dep:hnsw", "dep:rand", "dep:space", "dep:rand_pcg"]
[dev-dependencies]
fastrand = "2.0"
tempfile = "3.10.1"
criterion = { version = "0.8.1", features = ["html_reports"] }
[[example]]
name = "text_embedding"
required-features = ["vec"]
[[example]]
name = "text_embed_cache_bench"
required-features = ["vec"]
[[example]]
name = "openai_embedding"
required-features = ["api_embed"]
[[example]]
name = "simd_benchmark"
required-features = ["simd"]
[[bench]]
name = "search_precision_benchmark"
harness = false
[[bench]]
name = "vec_search_benchmark"
harness = false
required-features = ["hnsw_bench"]
+1 -1
View File
@@ -175,7 +175,7 @@
END OF TERMS AND CONDITIONS
Copyright 2024-2026 Memvid
Copyright 2026 Memvid
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
+236
View File
@@ -0,0 +1,236 @@
.PHONY: help build build-release test test-verbose clean fmt fmt-check clippy clippy-fix doc doc-open check install run-example docker-build docker-test docker-push pre-commit tree outdated bloat test-doc test-release package publish-dry-run coverage watch
# Default target
.DEFAULT_GOAL := help
# Variables
CARGO := cargo
RUST_VERSION := 1.85.0
FEATURES := lex,pdf_extract
# Colors for output
CYAN := \033[0;36m
GREEN := \033[0;32m
YELLOW := \033[0;33m
NC := \033[0m # No Color
help: ## Show this help message
@echo "$(CYAN)Memvid Makefile Commands:$(NC)"
@echo ""
@grep -E '^[a-zA-Z_-]+:.*?## .*$$' $(MAKEFILE_LIST) | sort | awk 'BEGIN {FS = ":.*?## "}; {printf " $(GREEN)%-20s$(NC) %s\n", $$1, $$2}'
@echo ""
install: ## Install Rust toolchain and dependencies
@echo "$(CYAN)Installing Rust toolchain...$(NC)"
@rustup toolchain install $(RUST_VERSION) || true
@rustup default $(RUST_VERSION) || true
@echo "$(CYAN)Installing cargo dependencies...$(NC)"
@$(CARGO) fetch
download-models: ## Download required models and dictionaries
@echo "$(CYAN)Downloading SymSpell dictionaries...$(NC)"
@mkdir -p data
@curl -L https://raw.githubusercontent.com/wolfgarbe/SymSpell/master/SymSpell/frequency_dictionary_en_82_765.txt -o data/frequency_dictionary_en_82_765.txt
@curl -L https://raw.githubusercontent.com/wolfgarbe/SymSpell/master/SymSpell/frequency_bigramdictionary_en_243_342.txt -o data/frequency_bigramdictionary_en_243_342.txt
check: ## Check code without building
@echo "$(CYAN)Checking code...$(NC)"
@$(CARGO) check --features $(FEATURES)
build: ## Build in debug mode
@echo "$(CYAN)Building in debug mode...$(NC)"
@$(CARGO) build --features $(FEATURES)
build-release: ## Build in release mode (optimized)
@echo "$(CYAN)Building in release mode...$(NC)"
@$(CARGO) build --release --features $(FEATURES)
build-verbose: ## Build with verbose output
@echo "$(CYAN)Building with verbose output...$(NC)"
@$(CARGO) build --verbose --features $(FEATURES)
build-release-verbose: ## Build release with verbose output
@echo "$(CYAN)Building release with verbose output...$(NC)"
@$(CARGO) build --release --verbose --features $(FEATURES)
build-all-features: ## Build with all features enabled
@echo "$(CYAN)Building with all features...$(NC)"
@$(CARGO) build --release --all-features
test: ## Run tests
@echo "$(CYAN)Running tests...$(NC)"
@$(CARGO) test --features $(FEATURES)
test-verbose: ## Run tests with output
@echo "$(CYAN)Running tests with output...$(NC)"
@$(CARGO) test --features $(FEATURES) -- --nocapture
test-release: ## Run tests in release mode
@echo "$(CYAN)Running tests in release mode...$(NC)"
@$(CARGO) test --release --features $(FEATURES)
test-doc: ## Run documentation tests only
@echo "$(CYAN)Running documentation tests...$(NC)"
@$(CARGO) test --doc --features $(FEATURES)
test-all-targets: ## Run all test targets (lib, bins, tests, examples)
@echo "$(CYAN)Running all test targets...$(NC)"
@$(CARGO) test --all-targets --features $(FEATURES)
test-no-fail-fast: ## Run all tests even if one fails
@echo "$(CYAN)Running all tests (no fail fast)...$(NC)"
@$(CARGO) test --features $(FEATURES) -- --no-fail-fast
test-integration: ## Run integration tests only
@echo "$(CYAN)Running integration tests...$(NC)"
@$(CARGO) test --test lifecycle --test search --test mutation --test crash_recovery --test doctor_recovery --test encryption_capsule --test replay_integrity --test single_file --features $(FEATURES)
test-unit: ## Run unit tests only
@echo "$(CYAN)Running unit tests...$(NC)"
@$(CARGO) test --lib --features $(FEATURES)
fmt: ## Format code
@echo "$(CYAN)Formatting code...$(NC)"
@$(CARGO) fmt --all
fmt-check: ## Check code formatting
@echo "$(CYAN)Checking code formatting...$(NC)"
@$(CARGO) fmt --all -- --check
clippy: ## Run clippy linter
@echo "$(CYAN)Running clippy...$(NC)"
@$(CARGO) clippy --all-targets --features $(FEATURES) -- -D warnings
clippy-fix: ## Run clippy and auto-fix issues
@echo "$(CYAN)Running clippy with auto-fix...$(NC)"
@$(CARGO) clippy --fix --all-targets --features $(FEATURES) -- -D warnings
doc: ## Generate documentation
@echo "$(CYAN)Generating documentation...$(NC)"
@$(CARGO) doc --features $(FEATURES) --no-deps
doc-open: ## Generate and open documentation
@echo "$(CYAN)Generating and opening documentation...$(NC)"
@$(CARGO) doc --features $(FEATURES) --no-deps --open
clean: ## Clean build artifacts
@echo "$(CYAN)Cleaning build artifacts...$(NC)"
@$(CARGO) clean
clean-all: clean ## Clean everything including target directory
@echo "$(CYAN)Cleaning all artifacts...$(NC)"
@rm -rf target/
run-example-basic: ## Run basic_usage example
@echo "$(CYAN)Running basic_usage example...$(NC)"
@$(CARGO) run --example basic_usage --features $(FEATURES)
run-example-pdf: ## Run pdf_ingestion example (usage: make run-example-pdf PDF_PATH=/path/to/pdf)
@if [ -z "$(PDF_PATH)" ]; then \
echo "$(YELLOW)Usage: make run-example-pdf PDF_PATH=/path/to/pdf$(NC)"; \
echo "$(YELLOW)Example: make run-example-pdf PDF_PATH=examples/1706.03762v7.pdf$(NC)"; \
exit 1; \
fi
@echo "$(CYAN)Running pdf_ingestion example with $(PDF_PATH)...$(NC)"
@$(CARGO) run --example pdf_ingestion --features $(FEATURES) -- $(PDF_PATH)
run-example-clip: ## Run clip_visual_search example (usage: make run-example-clip PDF_PATH=/path/to/pdf)
@if [ -z "$(PDF_PATH)" ]; then \
echo "$(YELLOW)Usage: make run-example-clip PDF_PATH=/path/to/pdf$(NC)"; \
echo "$(YELLOW)Example: make run-example-clip PDF_PATH=examples/document.pdf$(NC)"; \
exit 1; \
fi
@echo "$(CYAN)Running clip_visual_search example with $(PDF_PATH)...$(NC)"
@$(CARGO) run --example clip_visual_search --features $(FEATURES),clip -- $(PDF_PATH)
run-example-whisper: ## Run test_whisper example (usage: make run-example-whisper AUDIO_PATH=/path/to/audio)
@if [ -z "$(AUDIO_PATH)" ]; then \
echo "$(YELLOW)Usage: make run-example-whisper AUDIO_PATH=/path/to/audio$(NC)"; \
echo "$(YELLOW)Example: make run-example-whisper AUDIO_PATH=examples/call_sale.mp3$(NC)"; \
exit 1; \
fi
@echo "$(CYAN)Running test_whisper example with $(AUDIO_PATH)...$(NC)"
@$(CARGO) run --example test_whisper --features $(FEATURES),whisper -- $(AUDIO_PATH)
lint: fmt-check clippy ## Run all linting checks
verify: check lint test ## Run all verification checks (check, lint, test)
ci: verify build-release ## Run CI pipeline (verify + release build)
docker-build: ## Build Docker image
@echo "$(CYAN)Building Docker image...$(NC)"
@cd docker/cli && docker build -t memvid/cli:latest .
docker-test: ## Test Docker image
@echo "$(CYAN)Testing Docker image...$(NC)"
@cd docker/cli && ./test.sh
docker-push: ## Push Docker image to registry (requires login)
@echo "$(CYAN)Pushing Docker image...$(NC)"
@docker push memvid/cli:latest
docker-tag: ## Tag Docker image with version
@echo "$(CYAN)Tagging Docker image...$(NC)"
@VERSION=$$(grep "^version" Cargo.toml | cut -d'"' -f2); \
docker tag memvid/cli:latest memvid/cli:$$VERSION; \
echo "Tagged as memvid/cli:$$VERSION"
bench: ## Run benchmarks (if available)
@echo "$(CYAN)Running benchmarks...$(NC)"
@$(CARGO) bench --features $(FEATURES) || echo "$(YELLOW)No benchmarks found$(NC)"
update: ## Update dependencies
@echo "$(CYAN)Updating dependencies...$(NC)"
@$(CARGO) update
audit: ## Audit dependencies for security vulnerabilities
@echo "$(CYAN)Auditing dependencies...$(NC)"
@$(CARGO) audit || echo "$(YELLOW)cargo-audit not installed. Install with: cargo install cargo-audit$(NC)"
version: ## Show version information
@echo "$(CYAN)Version Information:$(NC)"
@$(CARGO) --version
@rustc --version
@echo ""
@echo "$(CYAN)Project version:$(NC)"
@grep "^version" Cargo.toml
tree: ## Show dependency tree
@echo "$(CYAN)Dependency tree:$(NC)"
@$(CARGO) tree --features $(FEATURES)
tree-duplicates: ## Show duplicate dependencies
@echo "$(CYAN)Duplicate dependencies:$(NC)"
@$(CARGO) tree --duplicates --features $(FEATURES)
outdated: ## Check for outdated dependencies
@echo "$(CYAN)Checking for outdated dependencies...$(NC)"
@$(CARGO) outdated || echo "$(YELLOW)cargo-outdated not installed. Install with: cargo install cargo-outdated$(NC)"
bloat: ## Analyze binary size bloat
@echo "$(CYAN)Analyzing binary size...$(NC)"
@$(CARGO) bloat --release --features $(FEATURES) || echo "$(YELLOW)cargo-bloat not installed. Install with: cargo install cargo-bloat$(NC)"
package: ## Create a package for publishing
@echo "$(CYAN)Creating package...$(NC)"
@$(CARGO) package
publish-dry-run: ## Dry run of publish (check if ready)
@echo "$(CYAN)Running publish dry-run...$(NC)"
@$(CARGO) publish --dry-run
coverage: ## Generate test coverage report
@echo "$(CYAN)Generating test coverage...$(NC)"
@$(CARGO) tarpaulin --features $(FEATURES) --out Html || echo "$(YELLOW)cargo-tarpaulin not installed. Install with: cargo install cargo-tarpaulin$(NC)"
watch: ## Watch for changes and run tests
@echo "$(CYAN)Watching for changes...$(NC)"
@$(CARGO) watch -x "test --features $(FEATURES)" || echo "$(YELLOW)cargo-watch not installed. Install with: cargo install cargo-watch$(NC)"
watch-build: ## Watch for changes and rebuild
@echo "$(CYAN)Watching for changes and rebuilding...$(NC)"
@$(CARGO) watch -x "build --features $(FEATURES)" || echo "$(YELLOW)cargo-watch not installed. Install with: cargo install cargo-watch$(NC)"
pre-commit: fmt-check clippy test ## Run pre-commit checks (fmt, clippy, test)
+249 -58
View File
@@ -1,12 +1,21 @@
<img width="2000" height="491" alt="Social Cover (6)" src="https://github.com/user-attachments/assets/4e256804-53ac-4173-bcff-81994d52bf5c" />
<!-- HEADER:START -->
<img width="2000" height="524" alt="Social Cover (9)"
src="https://github.com/user-attachments/assets/cf66f045-c8be-494b-b696-b8d7e4fb709c" />
<!-- HEADER:END -->
<div style="height: 16px;"></div>
<p align="center">
<a href="https://trendshift.io/repositories/17293" target="_blank"><img src="https://trendshift.io/api/badge/repositories/17293" alt="memvid%2Fmemvid | Trendshift" style="width: 250px; height: 55px;" width="250" height="55"/></a>
</p>
<!-- BADGES:END -->
<p align="center">
<strong>Memvid is a single-file memory layer for AI agents with instant retrieval and long-term memory.</strong><br/>
Persistent, versioned, and portable memory, without databases.
</p>
<!-- NAV:START -->
<p align="center">
<a href="https://www.memvid.com">Website</a>
·
@@ -16,11 +25,12 @@
·
<a href="https://github.com/memvid/memvid/discussions">Discussions</a>
</p>
<!-- NAV:END -->
<!-- BADGES:START -->
<p align="center">
<a href="https://crates.io/crates/memvid-core"><img src="https://img.shields.io/crates/v/memvid-core?style=flat-square&logo=rust" alt="Crates.io" /></a>
<a href="https://docs.rs/memvid-core"><img src="https://img.shields.io/docsrs/memvid-core?style=flat-square&logo=docs.rs" alt="docs.rs" /></a>
<a href="https://github.com/memvid/memvid/actions"><img src="https://img.shields.io/github/actions/workflow/status/memvid/memvid/ci.yml?style=flat-square&logo=github" alt="CI" /></a>
<a href="https://github.com/memvid/memvid/blob/main/LICENSE"><img src="https://img.shields.io/badge/license-Apache%202.0-blue?style=flat-square" alt="License" /></a>
</p>
@@ -28,11 +38,19 @@
<a href="https://github.com/memvid/memvid/stargazers"><img src="https://img.shields.io/github/stars/memvid/memvid?style=flat-square&logo=github" alt="Stars" /></a>
<a href="https://github.com/memvid/memvid/network/members"><img src="https://img.shields.io/github/forks/memvid/memvid?style=flat-square&logo=github" alt="Forks" /></a>
<a href="https://github.com/memvid/memvid/issues"><img src="https://img.shields.io/github/issues/memvid/memvid?style=flat-square&logo=github" alt="Issues" /></a>
<a href="https://discord.gg/memvid"><img src="https://img.shields.io/discord/1234567890?style=flat-square&logo=discord&label=discord" alt="Discord" /></a>
<a href="https://discord.gg/7RUve6Zrrv"><img src="https://img.shields.io/discord/1442910055233224745?style=flat-square&logo=discord&label=discord" alt="Discord" /></a>
</p>
<h2 align="center">⭐️ Leave a STAR to support the project ⭐️</h2>
</p>
## Benchmark Highlights
**🚀 Higher accuracy than any other memory system :** +35% SOTA on LoCoMo, best-in-class long-horizon conversational recall & reasoning
**🧠 Superior multi-hop & temporal reasoning:** +76% multi-hop, +56% temporal vs. the industry average
**⚡ Ultra-low latency at scale** 0.025ms P50 and 0.075ms P99, with 1,372× higher throughput than standard
**🔬 Fully reproducible benchmarks:** LoCoMo (10 × ~26K-token conversations), open-source eval, LLM-as-Judge
## What is Memvid?
@@ -43,9 +61,8 @@ Instead of running complex RAG pipelines or server-based vector databases, Memvi
The result is a model-agnostic, infrastructure-free memory layer that gives AI agents persistent, long-term memory they can carry anywhere.
---
## Why Video Frames?
## What are Smart Frames?
Memvid draws inspiration from video encoding, not to store video, but to **organize AI memory as an append-only, ultra-efficient sequence of Smart Frames.**
@@ -54,62 +71,60 @@ Frames are grouped in a way that allows efficient compression, indexing, and par
This frame-based design enables:
- Append-only writes without modifying or corrupting existing data
- Queries over past memory states
- Timeline-style inspection of how knowledge evolves
- Crash safety through committed, immutable frames
- Efficient compression using techniques adapted from video encoding
- Append-only writes without modifying or corrupting existing data
- Queries over past memory states
- Timeline-style inspection of how knowledge evolves
- Crash safety through committed, immutable frames
- Efficient compression using techniques adapted from video encoding
The result is a single file that behaves like a rewindable memory timeline for AI systems.
---
## Core Concepts
- **Living Memory Engine**
Continuously append, branch, and evolve memory across sessions.
- **Living Memory Engine**
Continuously append, branch, and evolve memory across sessions.
- **Capsule Context (`.mv2`)**
Self-contained, shareable memory capsules with rules and expiry.
- **Capsule Context (`.mv2`)**
Self-contained, shareable memory capsules with rules and expiry.
- **Time-Travel Debugging**
Rewind, replay, or branch any memory state.
- **Time-Travel Debugging**
Rewind, replay, or branch any memory state.
- **Smart Recall**
Sub-5ms local memory access with predictive caching.
- **Smart Recall**
Sub-5ms local memory access with predictive caching.
- **Codec Intelligence**
Auto-selects and upgrades compression over time.
- **Codec Intelligence**
Auto-selects and upgrades compression over time.
---
## Use Cases
Memvid is a portable, serverless memory layer that gives AI agents persistent memory and fast recall. Because it's model-agnostic, multi-modal, and works fully offline, developers are using Memvid across a wide range of real-world applications.
- Long-Running AI Agents
- Enterprise Knowledge Bases
- Offline-First AI Systems
- Codebase Understanding
- Customer Support Agents
- Workflow Automation
- Sales and Marketing Copilots
- Personal Knowledge Assistants
- Medical, Legal, and Financial Agents
- Auditable and Debuggable AI Workflows
- Custom Applications
- Long-Running AI Agents
- Enterprise Knowledge Bases
- Offline-First AI Systems
- Codebase Understanding
- Customer Support Agents
- Workflow Automation
- Sales and Marketing Copilots
- Personal Knowledge Assistants
- Medical, Legal, and Financial Agents
- Auditable and Debuggable AI Workflows
- Custom Applications
---
## SDKs & CLI
Use Memvid in your preferred language:
| Package | Install | Links |
|---------|---------|-------|
| **CLI** | `npm install -g memvid-cli` | [![npm](https://img.shields.io/npm/v/memvid-cli?style=flat-square)](https://www.npmjs.com/package/memvid-cli) |
| **Node.js SDK** | `npm install @memvid/sdk` | [![npm](https://img.shields.io/npm/v/@memvid/sdk?style=flat-square)](https://www.npmjs.com/package/@memvid/sdk) |
| **Python SDK** | `pip install memvid-sdk` | [![PyPI](https://img.shields.io/pypi/v/memvid-sdk?style=flat-square)](https://pypi.org/project/memvid-sdk/) |
| **Rust** | `cargo add memvid-core` | [![Crates.io](https://img.shields.io/crates/v/memvid-core?style=flat-square)](https://crates.io/crates/memvid-core) |
| Package | Install | Links |
| --------------- | --------------------------- | ------------------------------------------------------------------------------------------------------------------- |
| **CLI** | `npm install -g memvid-cli` | [![npm](https://img.shields.io/npm/v/memvid-cli?style=flat-square)](https://www.npmjs.com/package/memvid-cli) |
| **Node.js SDK** | `npm install @memvid/sdk` | [![npm](https://img.shields.io/npm/v/@memvid/sdk?style=flat-square)](https://www.npmjs.com/package/@memvid/sdk) |
| **Python SDK** | `pip install memvid-sdk` | [![PyPI](https://img.shields.io/pypi/v/memvid-sdk?style=flat-square)](https://pypi.org/project/memvid-sdk/) |
| **Rust** | `cargo add memvid-core` | [![Crates.io](https://img.shields.io/crates/v/memvid-core?style=flat-square)](https://crates.io/crates/memvid-core) |
---
@@ -117,7 +132,7 @@ Use Memvid in your preferred language:
### Requirements
- **Rust 1.85.0+** — Install from [rustup.rs](https://rustup.rs)
- **Rust 1.85.0+** — Install from [rustup.rs](https://rustup.rs)
### Add to Your Project
@@ -128,16 +143,18 @@ memvid-core = "2.0"
### Feature Flags
| Feature | Description |
|---------|-------------|
| `lex` | Full-text search with BM25 ranking (Tantivy) |
| `pdf_extract` | Pure Rust PDF text extraction |
| `vec` | Vector similarity search (HNSW + ONNX) |
| `clip` | CLIP visual embeddings for image search |
| `whisper` | Audio transcription with Whisper |
| `temporal_track` | Natural language date parsing ("last Tuesday") |
| `parallel_segments` | Multi-threaded ingestion |
| `encryption` | Password-based encryption capsules (.mv2e) |
| Feature | Description |
| ------------------- | ---------------------------------------------------------------- |
| `lex` | Full-text search with BM25 ranking (Tantivy) |
| `pdf_extract` | Pure Rust PDF text extraction |
| `vec` | Vector similarity search (HNSW + local text embeddings via ONNX) |
| `clip` | CLIP visual embeddings for image search |
| `whisper` | Audio transcription with Whisper |
| `api_embed` | Cloud API embeddings (OpenAI) |
| `temporal_track` | Natural language date parsing ("last Tuesday") |
| `parallel_segments` | Multi-threaded ingestion |
| `encryption` | Password-based encryption capsules (.mv2e) |
| `symspell_cleanup` | Robust PDF text repair (fixes "emp lo yee" -> "employee") |
Enable features as needed:
@@ -146,7 +163,6 @@ Enable features as needed:
memvid-core = { version = "2.0", features = ["lex", "vec", "temporal_track"] }
```
---
## Quick Start
@@ -276,10 +292,177 @@ cargo run --example clip_visual_search --features clip
Audio transcription (requires `whisper` feature):
```bash
cargo run --example test_whisper --features whisper
cargo run --example test_whisper --features whisper -- /path/to/audio.mp3
```
---
**Available Models:**
| Model | Size | Speed | Use Case |
| --------------------- | ------ | ------- | ----------------------------------- |
| `whisper-small-en` | 244 MB | Slowest | Best accuracy (default) |
| `whisper-tiny-en` | 75 MB | Fast | Balanced |
| `whisper-tiny-en-q8k` | 19 MB | Fastest | Quick testing, resource-constrained |
**Model Selection:**
```bash
# Default (FP32 small, highest accuracy)
cargo run --example test_whisper --features whisper -- audio.mp3
# Quantized tiny (75% smaller, faster)
MEMVID_WHISPER_MODEL=whisper-tiny-en-q8k cargo run --example test_whisper --features whisper -- audio.mp3
```
**Programmatic Configuration:**
```rust
use memvid_core::{WhisperConfig, WhisperTranscriber};
// Default FP32 small model
let config = WhisperConfig::default();
// Quantized tiny model (faster, smaller)
let config = WhisperConfig::with_quantization();
// Specific model
let config = WhisperConfig::with_model("whisper-tiny-en-q8k");
let transcriber = WhisperTranscriber::new(&config)?;
let result = transcriber.transcribe_file("audio.mp3")?;
println!("{}", result.text);
```
## Text Embedding Models
The `vec` feature includes local text embedding support using ONNX models. Before using local text embeddings, you need to download the model files manually.
### Quick Start: BGE-small (Recommended)
Download the default BGE-small model (384 dimensions, fast and efficient):
```bash
mkdir -p ~/.cache/memvid/text-models
# Download ONNX model
curl -L 'https://huggingface.co/BAAI/bge-small-en-v1.5/resolve/main/onnx/model.onnx' \
-o ~/.cache/memvid/text-models/bge-small-en-v1.5.onnx
# Download tokenizer
curl -L 'https://huggingface.co/BAAI/bge-small-en-v1.5/resolve/main/tokenizer.json' \
-o ~/.cache/memvid/text-models/bge-small-en-v1.5_tokenizer.json
```
### Available Models
| Model | Dimensions | Size | Best For |
| ----------------------- | ---------- | ------ | --------------- |
| `bge-small-en-v1.5` | 384 | ~120MB | Default, fast |
| `bge-base-en-v1.5` | 768 | ~420MB | Better quality |
| `nomic-embed-text-v1.5` | 768 | ~530MB | Versatile tasks |
| `gte-large` | 1024 | ~1.3GB | Highest quality |
### Other Models
**BGE-base** (768 dimensions):
```bash
curl -L 'https://huggingface.co/BAAI/bge-base-en-v1.5/resolve/main/onnx/model.onnx' \
-o ~/.cache/memvid/text-models/bge-base-en-v1.5.onnx
curl -L 'https://huggingface.co/BAAI/bge-base-en-v1.5/resolve/main/tokenizer.json' \
-o ~/.cache/memvid/text-models/bge-base-en-v1.5_tokenizer.json
```
**Nomic** (768 dimensions):
```bash
curl -L 'https://huggingface.co/nomic-ai/nomic-embed-text-v1.5/resolve/main/onnx/model.onnx' \
-o ~/.cache/memvid/text-models/nomic-embed-text-v1.5.onnx
curl -L 'https://huggingface.co/nomic-ai/nomic-embed-text-v1.5/resolve/main/tokenizer.json' \
-o ~/.cache/memvid/text-models/nomic-embed-text-v1.5_tokenizer.json
```
**GTE-large** (1024 dimensions):
```bash
curl -L 'https://huggingface.co/thenlper/gte-large/resolve/main/onnx/model.onnx' \
-o ~/.cache/memvid/text-models/gte-large.onnx
curl -L 'https://huggingface.co/thenlper/gte-large/resolve/main/tokenizer.json' \
-o ~/.cache/memvid/text-models/gte-large_tokenizer.json
```
### Usage in Code
```rust
use memvid_core::text_embed::{LocalTextEmbedder, TextEmbedConfig};
use memvid_core::types::embedding::EmbeddingProvider;
// Use default model (BGE-small)
let config = TextEmbedConfig::default();
let embedder = LocalTextEmbedder::new(config)?;
let embedding = embedder.embed_text("hello world")?;
assert_eq!(embedding.len(), 384);
// Use different model
let config = TextEmbedConfig::bge_base();
let embedder = LocalTextEmbedder::new(config)?;
```
See `examples/text_embedding.rs` for a complete example with similarity computation and search ranking.
### Model Consistency
To prevent accidental model mixing (e.g., querying a BGE-small index with OpenAI embeddings), you can explicitly bind your Memvid instance to a specific model name:
```rust
// Bind the index to a specific model.
// If the index was previously created with a different model, this will return an error.
mem.set_vec_model("bge-small-en-v1.5")?;
```
This binding is persistent. Once set, future attempts to use a different model name will fail fast with a `ModelMismatch` error.
## API Embeddings (OpenAI)
The `api_embed` feature enables cloud-based embedding generation using OpenAI's API.
### Setup
Set your OpenAI API key:
```bash
export OPENAI_API_KEY="sk-..."
```
### Usage
```rust
use memvid_core::api_embed::{OpenAIConfig, OpenAIEmbedder};
use memvid_core::types::embedding::EmbeddingProvider;
// Use default model (text-embedding-3-small)
let config = OpenAIConfig::default();
let embedder = OpenAIEmbedder::new(config)?;
let embedding = embedder.embed_text("hello world")?;
assert_eq!(embedding.len(), 1536);
// Use higher quality model
let config = OpenAIConfig::large(); // text-embedding-3-large (3072 dims)
let embedder = OpenAIEmbedder::new(config)?;
```
### Available Models
| Model | Dimensions | Best For |
| ------------------------ | ---------- | -------------------------- |
| `text-embedding-3-small` | 1536 | Default, fastest, cheapest |
| `text-embedding-3-large` | 3072 | Highest quality |
| `text-embedding-ada-002` | 1536 | Legacy model |
See `examples/openai_embedding.rs` for a complete example.
## File Format
@@ -307,7 +490,7 @@ No `.wal`, `.lock`, `.shm`, or sidecar files. Ever.
See [MV2_SPEC.md](MV2_SPEC.md) for the complete file format specification.
---
## Support
@@ -318,6 +501,14 @@ Email: contact@memvid.com
---
> **Memvid v1 (QR-based memory) is deprecated**
>
> If you are referencing QR codes, you are using outdated information.
>
> See: https://docs.memvid.com/memvid-v1-deprecation
---
## License
Apache License 2.0 — see the [LICENSE](LICENSE) file for details.
+176
View File
@@ -0,0 +1,176 @@
//! Search precision benchmarks for implicit AND operator change.
//!
//! This benchmark suite measures the performance impact of changing the implicit
//! query operator from OR to AND. It verifies that the precision improvement
//! (33% → 100%) comes with no query latency regression.
//!
//! # Benchmarks
//!
//! - `query_two_words`: Measures latency for simple two-word queries
//! - `precision_calculation`: Measures precision metrics and filtering overhead
//! - `result_count`: Measures result set size impact
//!
//! # Running
//!
//! ```bash
//! cargo bench --bench search_precision_benchmark --features lex
//! ```
use criterion::{Criterion, criterion_group, criterion_main};
use memvid_core::{Memvid, PutOptions, SearchRequest};
use std::time::Instant;
/// Setup test corpus
fn setup_corpus(size: usize) -> std::path::PathBuf {
let temp_file = std::env::temp_dir().join(format!("bench_{}.mv2", size));
let _ = std::fs::remove_file(&temp_file);
let mut mem = Memvid::create(&temp_file).unwrap();
let topics = [
"machine learning neural networks",
"python programming development",
"machine learning with python",
"rust systems programming",
"web development javascript",
];
for i in 0..size {
let content = format!("Document {} about {}", i, topics[i % topics.len()]);
mem.put_bytes_with_options(
content.as_bytes(),
PutOptions::builder().title(format!("Doc {}", i)).build(),
)
.unwrap();
if (i + 1) % 100 == 0 {
mem.commit().unwrap();
}
}
mem.commit().unwrap();
temp_file
}
fn bench_query_latency(c: &mut Criterion) {
let corpus_path = setup_corpus(1000);
c.bench_function("query_two_words", |b| {
b.iter_custom(|iters| {
let mut total = std::time::Duration::ZERO;
for _ in 0..iters {
let mut mem = Memvid::open(&corpus_path).unwrap(); // FIX: mut
let start = Instant::now();
let _results = mem
.search(SearchRequest {
query: "machine learning".to_string(),
top_k: 10,
snippet_chars: 200,
uri: None,
scope: None,
cursor: None,
#[cfg(feature = "temporal_track")]
temporal: None,
as_of_frame: None,
as_of_ts: None,
no_sketch: false,
acl_context: None,
acl_enforcement_mode: memvid_core::types::AclEnforcementMode::Audit,
})
.unwrap();
total += start.elapsed();
}
total
});
});
std::fs::remove_file(&corpus_path).ok();
}
fn bench_precision(c: &mut Criterion) {
let corpus_path = setup_corpus(1000);
c.bench_function("precision_calculation", |b| {
b.iter_custom(|iters| {
let mut total = std::time::Duration::ZERO;
for _ in 0..iters {
let mut mem = Memvid::open(&corpus_path).unwrap(); // FIX: mut
let start = Instant::now();
let results = mem
.search(SearchRequest {
query: "machine python".to_string(),
top_k: 100,
snippet_chars: 200,
uri: None,
scope: None,
cursor: None,
#[cfg(feature = "temporal_track")]
temporal: None,
as_of_frame: None,
as_of_ts: None,
no_sketch: false,
acl_context: None,
acl_enforcement_mode: memvid_core::types::AclEnforcementMode::Audit,
})
.unwrap();
let _relevant = results
.hits
.iter()
.filter(|hit| {
let text = hit.text.to_lowercase();
text.contains("machine") && text.contains("python")
})
.count();
total += start.elapsed();
}
total
});
});
std::fs::remove_file(&corpus_path).ok();
}
fn bench_result_count(c: &mut Criterion) {
let corpus_path = setup_corpus(1000);
c.bench_function("result_count", |b| {
b.iter_custom(|iters| {
let mut total = std::time::Duration::ZERO;
for _ in 0..iters {
let mut mem = Memvid::open(&corpus_path).unwrap(); // FIX: mut
let start = Instant::now();
let results = mem
.search(SearchRequest {
query: "machine learning".to_string(),
top_k: 100,
snippet_chars: 200,
uri: None,
scope: None,
cursor: None,
#[cfg(feature = "temporal_track")]
temporal: None,
as_of_frame: None,
as_of_ts: None,
no_sketch: false,
acl_context: None,
acl_enforcement_mode: memvid_core::types::AclEnforcementMode::Audit,
})
.unwrap();
let _count = results.hits.len();
total += start.elapsed();
}
total
});
});
std::fs::remove_file(&corpus_path).ok();
}
criterion_group!(
benches,
bench_query_latency,
bench_precision,
bench_result_count
);
criterion_main!(benches);
+145
View File
@@ -0,0 +1,145 @@
use criterion::{Criterion, black_box, criterion_group, criterion_main};
use memvid_core::types::FrameId;
use memvid_core::vec::{VecDocument, VecIndex, VecIndexBuilder};
fn generate_vectors(count: usize, dim: usize) -> Vec<Vec<f32>> {
let mut vectors = Vec::with_capacity(count);
for _ in 0..count {
let mut vec = Vec::with_capacity(dim);
for _ in 0..dim {
vec.push(fastrand::f32());
}
vectors.push(vec);
}
vectors
}
fn bench_search_10k(c: &mut Criterion) {
let count = 10_000;
let dim = 128; // Smaller dimension for faster setup in benchmarks
let vectors = generate_vectors(count, dim);
let query = generate_vectors(1, dim).pop().unwrap();
// Build HNSW Index (via Builder which triggers HNSW for > 1000)
let mut builder = VecIndexBuilder::new();
for (i, vec) in vectors.iter().enumerate() {
builder.add_document(i as FrameId, vec.clone());
}
let artifact = builder.finish().expect("finish hnsw");
let hnsw_index = VecIndex::decode(&artifact.bytes).expect("decode hnsw");
// Build Brute Force Index (Force Uncompressed)
let documents: Vec<VecDocument> = vectors
.iter()
.enumerate()
.map(|(i, vec)| VecDocument {
frame_id: i as FrameId,
embedding: vec.clone(),
})
.collect();
let brute_index = VecIndex::Uncompressed { documents };
let mut group = c.benchmark_group("search_10k");
group.bench_function("hnsw", |b| {
b.iter(|| {
hnsw_index.search(black_box(&query), black_box(10));
})
});
group.bench_function("brute_force", |b| {
b.iter(|| {
brute_index.search(black_box(&query), black_box(10));
})
});
group.finish();
}
fn bench_search_50k(c: &mut Criterion) {
let count = 50_000;
let dim = 128;
let vectors = generate_vectors(count, dim);
let query = generate_vectors(1, dim).pop().unwrap();
let mut builder = VecIndexBuilder::new();
for (i, vec) in vectors.iter().enumerate() {
builder.add_document(i as FrameId, vec.clone());
}
let artifact = builder.finish().expect("finish hnsw");
let hnsw_index = VecIndex::decode(&artifact.bytes).expect("decode hnsw");
let documents: Vec<VecDocument> = vectors
.iter()
.enumerate()
.map(|(i, vec)| VecDocument {
frame_id: i as FrameId,
embedding: vec.clone(),
})
.collect();
let brute_index = VecIndex::Uncompressed { documents };
let mut group = c.benchmark_group("search_50k");
group.bench_function("hnsw", |b| {
b.iter(|| {
hnsw_index.search(black_box(&query), black_box(10));
})
});
group.bench_function("brute_force", |b| {
b.iter(|| {
brute_index.search(black_box(&query), black_box(10));
})
});
group.finish();
}
fn bench_search_100k(c: &mut Criterion) {
let count = 100_000;
let dim = 128;
let vectors = generate_vectors(count, dim);
let query = generate_vectors(1, dim).pop().unwrap();
let mut builder = VecIndexBuilder::new();
for (i, vec) in vectors.iter().enumerate() {
builder.add_document(i as FrameId, vec.clone());
}
let artifact = builder.finish().expect("finish hnsw");
let hnsw_index = VecIndex::decode(&artifact.bytes).expect("decode hnsw");
let documents: Vec<VecDocument> = vectors
.iter()
.enumerate()
.map(|(i, vec)| VecDocument {
frame_id: i as FrameId,
embedding: vec.clone(),
})
.collect();
let brute_index = VecIndex::Uncompressed { documents };
let mut group = c.benchmark_group("search_100k");
group.bench_function("hnsw", |b| {
b.iter(|| {
let _ = hnsw_index.search(black_box(&query), black_box(10));
})
});
group.bench_function("brute_force", |b| {
b.iter(|| {
let _ = brute_index.search(black_box(&query), black_box(10));
})
});
group.finish();
}
criterion_group!(
benches,
bench_search_10k,
bench_search_50k,
bench_search_100k
);
criterion_main!(benches);
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
+82
View File
@@ -0,0 +1,82 @@
# Docker Images for Memvid
This directory contains Docker configurations for Memvid components.
## Available Images
### Memvid Core (`core/`)
The Memvid Core Docker images provide containerized Rust development, testing, and build environments for the `memvid-core` library.
**Quick Start:**
```bash
# Development environment
cd core
docker-compose up -d dev
docker-compose exec dev bash
# Run tests
docker-compose run --rm test
# Build release
docker-compose run --rm build
```
For detailed usage, see [core/README.md](core/README.md).
### Memvid CLI (`cli/`)
The Memvid CLI Docker image provides a containerized version of the `memvid-cli` tool, allowing you to run Memvid commands without installing Node.js or dealing with platform-specific binaries.
**Quick Start:**
```bash
# Pull the image
docker pull memvid/cli
# Create a memory
docker run --rm -v $(pwd):/data memvid/cli create my-memory.mv2
# Add documents
docker run --rm -v $(pwd):/data memvid/cli put my-memory.mv2 --input doc.pdf
# Search
docker run --rm -v $(pwd):/data memvid/cli find my-memory.mv2 --query "search"
```
For detailed usage instructions, examples, and Docker Compose configurations, see [cli/README.md](cli/README.md).
## Building Images
### Build CLI Image Locally
```bash
cd cli
docker build -t memvid/cli:test .
```
## Publishing
Docker images are automatically built and published to Docker Hub via GitHub Actions when tags are pushed. See `.github/workflows/docker-release.yml` for the CI/CD configuration.
**Image Registry:**
- Docker Hub: `memvid/cli`
- Tags: `latest`, `2.0.129`, and version-specific tags
## Architecture Support
The CLI image supports multi-architecture builds:
- `linux/amd64`
- `linux/arm64`
## Security
The CLI image runs as a non-root user (`memvid`) for improved security. When mounting volumes, ensure your host directories have appropriate permissions.
## Links
- [Core Documentation](core/README.md)
- [CLI Documentation](cli/README.md)
- [CLI Testing Guide](cli/TESTING.md)
- [Main Project README](../README.md)
- [Memvid Documentation](https://docs.memvid.com)
+7
View File
@@ -0,0 +1,7 @@
node_modules
npm-debug.log
.git
.gitignore
*.md
.env
.DS_Store
+33
View File
@@ -0,0 +1,33 @@
# Memvid CLI Docker Image
FROM ubuntu:24.04
LABEL org.opencontainers.image.title="Memvid CLI"
LABEL org.opencontainers.image.description="AI memory CLI with semantic search"
LABEL org.opencontainers.image.url="https://memvid.com"
LABEL org.opencontainers.image.source="https://github.com/memvid/memvid"
LABEL org.opencontainers.image.vendor="Memvid"
LABEL org.opencontainers.image.version="2.0.131"
# Install system deps + Node.js 24
RUN apt-get update && apt-get install -y --no-install-recommends \
ca-certificates \
curl \
libssl3 \
&& curl -fsSL https://deb.nodesource.com/setup_24.x | bash - \
&& apt-get install -y --no-install-recommends nodejs \
&& rm -rf /var/lib/apt/lists/*
# Install memvid CLI (pinned version)
RUN npm install -g memvid-cli@latest \
&& npm cache clean --force
# Create non-root user (recommended)
RUN useradd -m memvid
USER memvid
# Working directory for memory files
WORKDIR /data
# Entrypoint
ENTRYPOINT ["memvid"]
CMD ["--help"]
+101
View File
@@ -0,0 +1,101 @@
# Memvid CLI Docker Image
AI memory CLI with crash-safe, single-file storage and semantic search.
## Quick Start
```bash
# Pull the image
docker pull memvid/cli
# Create a memory
docker run --rm -v $(pwd):/data memvid/cli create my-memory.mv2
# Add documents
docker run --rm -v $(pwd):/data memvid/cli put my-memory.mv2 --input doc.pdf
# Search
docker run --rm -v $(pwd):/data memvid/cli find my-memory.mv2 --query "search"
```
## Basic Commands
```bash
# Show help
docker run --rm memvid/cli
# Show version
docker run --rm memvid/cli --version
# Create a memory file (mount local directory)
docker run --rm -v $(pwd):/data memvid/cli create my-memory.mv2
# Ingest a document
docker run --rm -v $(pwd):/data memvid/cli put my-memory.mv2 --input document.pdf
# Search the memory
docker run --rm -v $(pwd):/data memvid/cli find my-memory.mv2 --query "search term"
# Ask questions (requires API key for LLM)
docker run --rm -v $(pwd):/data \
-e OPENAI_API_KEY="sk-..." \
memvid/cli ask my-memory.mv2 "What is this about?" -m openai
# View stats
docker run --rm -v $(pwd):/data memvid/cli stats my-memory.mv2
```
## With API Keys
```bash
# Pass Memvid API key for cloud features
docker run --rm -v $(pwd):/data \
-e MEMVID_API_KEY="mv2_..." \
-e OPENAI_API_KEY="sk-..." \
memvid/cli ask my-memory.mv2 "your question"
```
## Shell Alias (Recommended)
Add to `~/.bashrc` or `~/.zshrc`:
```bash
alias memvid='docker run --rm -v $(pwd):/data -e MEMVID_API_KEY -e OPENAI_API_KEY memvid/cli'
```
Then use normally:
```bash
memvid create my-memory.mv2
memvid put my-memory.mv2 --input docs/
memvid find my-memory.mv2 --query "hello"
```
## Docker Compose Example
```yaml
version: '3.8'
services:
memvid:
image: memvid/cli:latest
volumes:
- ./data:/data
environment:
- MEMVID_API_KEY=${MEMVID_API_KEY}
- OPENAI_API_KEY=${OPENAI_API_KEY}
entrypoint: ["memvid"]
command: ["stats", "my-memory.mv2"]
```
## Features
- Single-file `.mv2` storage
- Semantic + lexical search
- RAG question answering
- PDF, DOCX, images, audio support
## Links
- Website: https://memvid.com
- Docs: https://docs.memvid.com
- GitHub: https://github.com/memvid/memvid
+108
View File
@@ -0,0 +1,108 @@
# Testing Memvid CLI Docker Image
## Quick Test
### 1. Build the Image
```bash
cd docker/cli
docker build -t memvid/cli:test .
```
### 2. Test Basic Commands
```bash
# Test help command
docker run --rm memvid/cli:test --help
# Test version (if available)
docker run --rm memvid/cli:test --version
```
### 3. Test with Volume Mount
```bash
# Create a test directory
mkdir -p /tmp/memvid-test
cd /tmp/memvid-test
# Create a test document
echo "This is a test document about AI and machine learning." > test.txt
# Create a memory file
docker run --rm \
-v $(pwd):/data \
memvid/cli:test create test-memory.mv2
# Add the document
docker run --rm \
-v $(pwd):/data \
memvid/cli:test put test-memory.mv2 --input test.txt
# Search the memory
docker run --rm \
-v $(pwd):/data \
memvid/cli:test find test-memory.mv2 --query "AI"
# View stats
docker run --rm \
-v $(pwd):/data \
memvid/cli:test stats test-memory.mv2
```
### 4. Test with API Keys (if needed)
```bash
docker run --rm \
-v $(pwd):/data \
-e OPENAI_API_KEY="sk-..." \
memvid/cli:test ask test-memory.mv2 "What is this about?" -m openai
```
## Automated Testing
Run the test script:
```bash
cd docker/cli
./test.sh
```
## Multi-Architecture Testing
To test multi-arch builds locally (requires buildx):
```bash
# Create builder
docker buildx create --name memvid-builder --use
# Build for specific platform
docker buildx build \
--platform linux/amd64 \
--tag memvid/cli:test-amd64 \
--load \
.
# Test the amd64 image
docker run --rm memvid/cli:test-amd64 --help
```
## Troubleshooting
### Image not found
Make sure you've built the image:
```bash
docker build -t memvid/cli:test docker/cli
```
### Permission errors
Ensure Docker has permission to access the mounted directory:
```bash
chmod 755 /path/to/your/directory
```
### CLI not found
Verify the npm package exists:
```bash
docker run --rm memvid/cli:test which memvid
```
+44
View File
@@ -0,0 +1,44 @@
#!/bin/bash
# Test script for Memvid CLI Docker image
set -e
IMAGE_NAME="memvid/cli:test"
DOCKERFILE_DIR="$(cd "$(dirname "$0")" && pwd)"
echo "🐳 Building Docker image..."
docker build -t "$IMAGE_NAME" "$DOCKERFILE_DIR"
echo ""
echo "✅ Build complete! Testing basic commands..."
echo ""
echo "1️⃣ Testing help command..."
docker run --rm "$IMAGE_NAME" --help | head -5
echo ""
echo "2️⃣ Testing version command..."
docker run --rm "$IMAGE_NAME" --version || echo "Version command not available"
echo ""
echo "3️⃣ Testing with volume mount (create test memory)..."
TEST_DIR=$(mktemp -d)
cd "$TEST_DIR"
# Create a test document
echo "This is a test document about artificial intelligence and machine learning." > test.txt
docker run --rm \
-v "$TEST_DIR":/data \
"$IMAGE_NAME" create test-memory.mv2 || echo "Create command may require additional setup"
echo ""
echo "🧹 Cleaning up test directory..."
rm -rf "$TEST_DIR"
echo ""
echo "✅ Basic tests complete!"
echo ""
echo "To test manually, run:"
echo " docker run --rm $IMAGE_NAME --help"
echo " docker run --rm -v \$(pwd):/data $IMAGE_NAME create my-memory.mv2"
+215
View File
@@ -0,0 +1,215 @@
# Docker Setup for Memvid Core
This document describes how to use Docker with the Memvid Core Rust library.
## Quick Start
### Development Environment
```bash
# From project root or docker/core directory
cd docker/core
docker-compose up -d dev
# Enter the container
docker-compose exec dev bash
# Inside container, you can run:
cargo build
cargo test
cargo run --example basic_usage
```
### Run Tests
```bash
# Run all tests
cd docker/core
docker-compose run --rm test
# Or build test image manually (from project root)
docker build -f docker/core/Dockerfile.test -t memvid-test .
docker run --rm memvid-test
```
### Build Release
```bash
# Build release version
cd docker/core
docker-compose run --rm build
# Or build manually (from project root)
docker build -f docker/core/Dockerfile -t memvid-core:latest .
```
## Docker Images
### 1. Development Image (`Dockerfile.dev`)
Full development environment with all tools:
```bash
# From project root
docker build -f docker/core/Dockerfile.dev -t memvid-dev .
docker run -it --rm -v $(pwd):/app memvid-dev bash
```
**Features:**
- Rust toolchain 1.92
- All build dependencies
- Cargo watch (optional)
- Volume mounting for live development
### 2. Test Image (`Dockerfile.test`)
Optimized for running tests:
```bash
# From project root
docker build -f docker/core/Dockerfile.test -t memvid-test .
docker run --rm memvid-test
```
**Features:**
- Rust toolchain 1.92
- Test dependencies
- Runs tests automatically
### 3. Production Build (`Dockerfile`)
Multi-stage build for optimized production image:
```bash
# From project root
docker build -f docker/core/Dockerfile -t memvid-core:latest .
```
**Features:**
- Multi-stage build (smaller final image)
- Only runtime dependencies
- Optimized release build
## Docker Compose
### Services
- **`dev`** - Development environment with live code mounting
- **`test`** - Test runner
- **`build`** - Release builder
### Usage
```bash
# Start development environment
docker-compose up -d dev
# Run tests
docker-compose run --rm test
# Build release
docker-compose run --rm build
# Stop all services
docker-compose down
```
## Examples
### Run Examples in Docker
```bash
# Development container
docker-compose exec dev cargo run --example basic_usage
# With features
docker-compose exec dev cargo run --example pdf_ingestion --features lex,pdf_extract
```
### Memory-Constrained Testing
Test OOM prevention with memory limits:
```bash
# Test with memory limit (for OOM testing)
docker run --rm --memory=150m --memory-swap=150m \
-v $(pwd):/app \
memvid-test cargo test --features encryption --test encryption_capsule
```
### Build with Specific Features
```bash
# Build with all features
docker-compose exec dev cargo build --release --all-features
# Build with specific features
docker-compose exec dev cargo build --release --features lex,vec,encryption
```
## Volume Mounts
The docker-compose setup uses volumes for:
- **Source code** - Live mounting for development
- **Cargo cache** - Speeds up builds
- **Target cache** - Preserves build artifacts
## CI/CD Integration
### GitHub Actions Example
```yaml
name: Docker Build
on: [push, pull_request]
jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Build Docker image
run: docker build -t memvid-core:test .
- name: Run tests
run: docker run --rm memvid-core:test cargo test
```
## Troubleshooting
### Build Fails
```bash
# Clean build
docker-compose down -v
docker-compose build --no-cache
```
### Permission Issues
```bash
# Fix permissions
sudo chown -R $USER:$USER .
```
### Out of Memory
```bash
# Increase Docker memory limit in Docker Desktop settings
# Or use memory limits in docker run:
docker run --memory=2g --memory-swap=2g ...
```
## Best Practices
1. **Use docker-compose** for development
2. **Cache volumes** for faster builds
3. **Multi-stage builds** for production
4. **Test in containers** to match CI/CD environment
5. **Use .dockerignore** to exclude unnecessary files
## Related
- [CLI Docker Setup](../cli/README.md)
- [Docker Overview](../README.md)
- [Main Project README](../../README.md)
- [Contributing Guide](../../CONTRIBUTING.md)
+53
View File
@@ -0,0 +1,53 @@
# Multi-stage Dockerfile for Memvid Core (Rust Library)
# Build stage
FROM rust:1.92-slim-trixie AS builder
# Install build dependencies
RUN apt-get update && apt-get install -y --no-install-recommends \
pkg-config \
libssl-dev \
ca-certificates \
&& rm -rf /var/lib/apt/lists/*
WORKDIR /app
# Copy dependency files first for better caching
COPY Cargo.toml Cargo.lock ./
COPY rust-toolchain.toml ./
# Create a dummy src/lib.rs to build dependencies
RUN mkdir -p src && \
echo "// Dummy file for dependency building" > src/lib.rs && \
cargo build --release --features lex,pdf_extract && \
rm -rf src
# Copy source code
COPY src ./src
COPY examples ./examples
COPY tests ./tests
# Build the library with default features
RUN cargo build --release --features lex,pdf_extract
# Runtime stage - minimal image for running examples/tests
FROM debian:trixie-slim AS runtime
# Install runtime dependencies
RUN apt-get update && apt-get install -y --no-install-recommends \
ca-certificates \
libssl3 \
&& rm -rf /var/lib/apt/lists/*
WORKDIR /app
# Copy the built artifacts from builder
COPY --from=builder /app/target/release/examples ./examples
COPY --from=builder /app/target/release/deps ./deps
COPY --from=builder /app/target/release/libmemvid_core*.rlib ./lib/
# Set environment variables
ENV RUST_LOG=info
ENV PATH="/app/examples:${PATH}"
# Default command
CMD ["/bin/bash"]
+31
View File
@@ -0,0 +1,31 @@
# Development Dockerfile for Memvid Core
FROM rust:1.92-slim-trixie AS builder
# Install development dependencies
RUN apt-get update && apt-get install -y --no-install-recommends \
pkg-config \
libssl-dev \
ca-certificates \
git \
curl \
build-essential \
&& rm -rf /var/lib/apt/lists/*
WORKDIR /app
# Copy dependency files
COPY Cargo.toml Cargo.lock ./
COPY rust-toolchain.toml ./
# Install cargo-watch for development (optional)
RUN cargo install cargo-watch --locked || true
# Copy source code
COPY . .
# Set environment variables
ENV RUST_LOG=debug
ENV CARGO_TARGET_DIR=/app/target
# Default command (can be overridden)
CMD ["/bin/bash"]
+23
View File
@@ -0,0 +1,23 @@
# Test Dockerfile for Memvid Core
FROM rust:1.92-slim-trixie AS builder
# Install test dependencies
RUN apt-get update && apt-get install -y --no-install-recommends \
pkg-config \
libssl-dev \
ca-certificates \
git \
&& rm -rf /var/lib/apt/lists/*
WORKDIR /app
# Copy everything
COPY . .
# Set environment variables for testing
ENV RUST_BACKTRACE=1
ENV RUST_LOG=info
ENV CARGO_TARGET_DIR=/app/target
# Run tests with default features
CMD ["cargo", "test", "--features", "lex,pdf_extract", "--", "--nocapture"]
+215
View File
@@ -0,0 +1,215 @@
# Docker Setup for Memvid Core
This document describes how to use Docker with the Memvid Core Rust library.
## Quick Start
### Development Environment
```bash
# From project root or docker/core directory
cd docker/core
docker-compose up -d dev
# Enter the container
docker-compose exec dev bash
# Inside container, you can run:
cargo build
cargo test
cargo run --example basic_usage
```
### Run Tests
```bash
# Run all tests
cd docker/core
docker-compose run --rm test
# Or build test image manually (from project root)
docker build -f docker/core/Dockerfile.test -t memvid-test .
docker run --rm memvid-test
```
### Build Release
```bash
# Build release version
cd docker/core
docker-compose run --rm build
# Or build manually (from project root)
docker build -f docker/core/Dockerfile -t memvid-core:latest .
```
## Docker Images
### 1. Development Image (`Dockerfile.dev`)
Full development environment with all tools:
```bash
# From project root
docker build -f docker/core/Dockerfile.dev -t memvid-dev .
docker run -it --rm -v $(pwd):/app memvid-dev bash
```
**Features:**
- Rust toolchain 1.92
- All build dependencies
- Cargo watch (optional)
- Volume mounting for live development
### 2. Test Image (`Dockerfile.test`)
Optimized for running tests:
```bash
# From project root
docker build -f docker/core/Dockerfile.test -t memvid-test .
docker run --rm memvid-test
```
**Features:**
- Rust toolchain 1.92
- Test dependencies
- Runs tests automatically
### 3. Production Build (`Dockerfile`)
Multi-stage build for optimized production image:
```bash
# From project root
docker build -f docker/core/Dockerfile -t memvid-core:latest .
```
**Features:**
- Multi-stage build (smaller final image)
- Only runtime dependencies
- Optimized release build
## Docker Compose
### Services
- **`dev`** - Development environment with live code mounting
- **`test`** - Test runner
- **`build`** - Release builder
### Usage
```bash
# Start development environment
docker-compose up -d dev
# Run tests
docker-compose run --rm test
# Build release
docker-compose run --rm build
# Stop all services
docker-compose down
```
## Examples
### Run Examples in Docker
```bash
# Development container
docker-compose exec dev cargo run --example basic_usage
# With features
docker-compose exec dev cargo run --example pdf_ingestion --features lex,pdf_extract
```
### Memory-Constrained Testing
Test OOM prevention with memory limits:
```bash
# Test with memory limit (for OOM testing)
docker run --rm --memory=150m --memory-swap=150m \
-v $(pwd):/app \
memvid-test cargo test --features encryption --test encryption_capsule
```
### Build with Specific Features
```bash
# Build with all features
docker-compose exec dev cargo build --release --all-features
# Build with specific features
docker-compose exec dev cargo build --release --features lex,vec,encryption
```
## Volume Mounts
The docker-compose setup uses volumes for:
- **Source code** - Live mounting for development
- **Cargo cache** - Speeds up builds
- **Target cache** - Preserves build artifacts
## CI/CD Integration
### GitHub Actions Example
```yaml
name: Docker Build
on: [push, pull_request]
jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Build Docker image
run: docker build -t memvid-core:test .
- name: Run tests
run: docker run --rm memvid-core:test cargo test
```
## Troubleshooting
### Build Fails
```bash
# Clean build
docker-compose down -v
docker-compose build --no-cache
```
### Permission Issues
```bash
# Fix permissions
sudo chown -R $USER:$USER .
```
### Out of Memory
```bash
# Increase Docker memory limit in Docker Desktop settings
# Or use memory limits in docker run:
docker run --memory=2g --memory-swap=2g ...
```
## Best Practices
1. **Use docker-compose** for development
2. **Cache volumes** for faster builds
3. **Multi-stage builds** for production
4. **Test in containers** to match CI/CD environment
5. **Use .dockerignore** to exclude unnecessary files
## Related
- [CLI Docker Setup](../cli/README.md)
- [Docker Overview](../README.md)
- [Main Project README](../../README.md)
- [Contributing Guide](../../CONTRIBUTING.md)
+46
View File
@@ -0,0 +1,46 @@
services:
# Development environment
dev:
build:
context: ../..
dockerfile: docker/core/Dockerfile.dev
volumes:
- ../../:/app
- cargo-cache:/usr/local/cargo/registry
- target-cache:/app/target
environment:
- RUST_LOG=debug
- CARGO_TARGET_DIR=/app/target
stdin_open: true
tty: true
command: /bin/bash
# Test runner
test:
build:
context: ../..
dockerfile: docker/core/Dockerfile.test
volumes:
- ../../:/app
- cargo-cache:/usr/local/cargo/registry
- target-cache:/app/target
environment:
- RUST_BACKTRACE=1
- RUST_LOG=info
command: cargo test --features lex,pdf_extract -- --nocapture
# Build release
build:
build:
context: ../..
dockerfile: docker/core/Dockerfile
target: builder
volumes:
- ../../:/app
- cargo-cache:/usr/local/cargo/registry
- target-cache:/app/target
command: cargo build --release --features lex,pdf_extract
volumes:
cargo-cache:
target-cache:
+125
View File
@@ -0,0 +1,125 @@
# Contributing Translations (i18n)
Thank you for helping translate Memvids documentation and make the project accessible to a global audience.
This guide explains how to contribute translations for the main `README.md`.
---
## What Can Be Translated
- The main repository `README.md`
- Translations are stored in `docs/i18n/`
- Each language has a single translation file
---
## Translation Workflow
### 1. Check Existing Issues
Before starting:
- Check open issues to see if your language is already in progress
- If an issue exists, comment on it to indicate you want to work on it
- Only one contributor should work on a language at a time
---
### 2. Create a Translation Issue (If Needed)
If no issue exists for your language, open a new one using this format:
**Title:**
```
README translation: <Language> (<code>)
```
**Labels to apply:**
- `i18n`
- `documentation`
- `help wanted` or `good first issue`
---
### 3. Create the Translation File
1. Copy the main `README.md` from the repository root
2. Create a new file in `docs/i18n/` using this format:
```
README.<language-code>.md
```
**Examples:**
- `README.es.md`
- `README.zh-CN.md`
- `README.pt-BR.md`
Use standard ISO language codes (include region where applicable).
---
## Translation Guidelines
- **Preserve structure**
Keep headings, section order, and formatting identical to the original README
- **Preserve links and badges**
Do not modify URLs, badges, or shields
- **Preserve code blocks**
Keep code examples, commands, flags, and API names in English
- **Maintain technical accuracy**
Translate naturally, but do not change meaning or behavior
- **Language quality matters**
Native or fluent speakers are strongly preferred
---
## Submitting Your Translation
1. **Create a new branch:**
```bash
git checkout -b docs/i18n-<language-code>
```
2. **Add your translation file** under `docs/i18n/`
3. **Commit your changes:**
```bash
git commit -m "docs(i18n): add <Language> README translation"
```
4. **Push to your fork** and open a Pull Request
5. **Reference the translation issue** in your PR description
---
## Review Process
- Maintainers may request:
- Clarifications
- Formatting fixes
- Review by another native speaker
- Once approved, the PR will be merged
- The corresponding issue will be closed
---
## Keeping Translations Up to Date
When the main `README.md` changes significantly:
- Existing translations may need updates
- Contributors are encouraged to help keep translations in sync
---
## Getting Help
- Open an issue if you have questions about translations
- Use GitHub Discussions for coordination
- Contact: [contact@memvid.com](mailto:contact@memvid.com)
+160
View File
@@ -0,0 +1,160 @@
<!-- HEADER:START -->
<img width="2000" height="524" alt="Social Cover (9)" src="https://github.com/user-attachments/assets/cf66f045-c8be-494b-b696-b8d7e4fb709c" />
<!-- HEADER:END -->
<!-- FLAGS:START -->
<p align="center">
<a href="../../README.md">🇺🇸 English</a>
<a href="README.es.md">🇪🇸 Español</a>
<a href="README.fr.md">🇫🇷 Français</a>
<a href="README.so.md">🇸🇴 Soomaali</a>
<a href="README.ar.md">🇸🇦 العربية</a>
<a href="README.nl.md">🇧🇪/🇳🇱 Nederlands</a>
<a href="README.hi.md">🇮🇳 हिन्दी</a>
<a href="README.bn.md">🇧🇩 বাংলা</a>
<a href="README.cs.md">🇨🇿 Čeština</a>
<a href="README.ko.md">🇰🇷 한국어</a>
<a href="README.ja.md">🇯🇵 日本語</a>
<!-- Next Flag -->
</p>
<!-- FLAGS:END -->
<!-- NAV:START -->
<p align="center">
<a href="https://www.memvid.com">Website</a>
·
<a href="https://sandbox.memvid.com">Try Sandbox</a>
·
<a href="https://docs.memvid.com">Docs</a>
·
<a href="https://github.com/memvid/memvid/discussions">Discussions</a>
</p>
<!-- NAV:END -->
<!-- BADGES:START -->
<p align="center">
<a href="https://crates.io/crates/memvid-core"><img src="https://img.shields.io/crates/v/memvid-core?style=flat-square&logo=rust" alt="Crates.io" /></a>
<a href="https://docs.rs/memvid-core"><img src="https://img.shields.io/docsrs/memvid-core?style=flat-square&logo=docs.rs" alt="docs.rs" /></a>
<a href="https://github.com/memvid/memvid/blob/main/LICENSE"><img src="https://img.shields.io/badge/license-Apache%202.0-blue?style=flat-square" alt="License" /></a>
</p>
<p align="center">
<a href="https://github.com/memvid/memvid/stargazers"><img src="https://img.shields.io/github/stars/memvid/memvid?style=flat-square&logo=github" alt="Stars" /></a>
<a href="https://github.com/memvid/memvid/network/members"><img src="https://img.shields.io/github/forks/memvid/memvid?style=flat-square&logo=github" alt="Forks" /></a>
<a href="https://github.com/memvid/memvid/issues"><img src="https://img.shields.io/github/issues/memvid/memvid?style=flat-square&logo=github" alt="Issues" /></a>
<a href="https://discord.gg/2mynS7fcK7"><img src="https://img.shields.io/discord/1442910055233224745?style=flat-square&logo=discord&label=discord" alt="Discord" /></a>
</p>
<p align="center">
<a href="https://trendshift.io/repositories/17293" target="_blank"><img src="https://trendshift.io/api/badge/repositories/17293" alt="memvid%2Fmemvid | Trendshift" style="width: 250px; height: 55px;" width="250" height="55"/></a>
</p>
<!-- BADGES:END -->
# المساهمة في Memvid (الترجمة العربية)
<p align="center">
<strong>Memvid هي طبقة ذاكرة مكونة من ملف واحد لوكلاء الذكاء الاصطناعي (AI Agents)، توفر استرجاعاً فورياً وذاكرة طويلة المدى.</strong>
ذاكرة دائمة، مؤرشفة، وقابلة للنقل، دون الحاجة إلى قواعد بيانات.
</p>
<h2 align="center">⭐️ اترك نجمة (STAR) لدعم المشروع ⭐️</h2>
---
## ما هو Memvid؟
Memvid هو نظام ذاكرة محمول للذكاء الاصطناعي يقوم بتغليف بياناتك، والمتجهات (Embeddings)، وهيكل البحث، والبيانات الوصفية في **ملف واحد فقط**.
بدلاً من تشغيل خطوط أنابيب RAG معقدة أو قواعد بيانات متجهة تعتمد على الخادم، يتيح Memvid استرجاعاً سريعاً للبيانات مباشرة من الملف.
النتيجة هي طبقة ذاكرة مستقلة عن النموذج (Model-agnostic) ولا تحتاج إلى بنية تحتية، مما يمنح وكلاء الذكاء الاصطناعي ذاكرة دائمة وطويلة المدى يمكنهم حملها في أي مكان.
---
## لماذا "إطارات الفيديو" (Video Frames)؟
يستلهم Memvid فكرته من ترميز الفيديو، ليس لتخزين الفيديو، بل **لتنظيم ذاكرة الذكاء الاصطناعي كمتسلسلة فائقة الكفاءة من "الإطارات الذكية" (Smart Frames) التي تُضاف باستمرار.**
"الإطار الذكي" هو وحدة غير قابلة للتغيير تخزن المحتوى مع الطوابع الزمنية، والتحقق من البيانات (Checksums)، والبيانات الوصفية الأساسية. يتم تجميع هذه الإطارات بطريقة تسمح بضغط البيانات وفهرستها والقراءة المتوازية بكفاءة عالية.
يسمح هذا التصميم بـ:
- **إضافة البيانات فقط:** الكتابة دون تعديل أو إفساد البيانات الموجودة.
- **الاستعلام عبر الزمن:** البحث في حالات الذاكرة السابقة.
- **جدول زمني للمعرفة:** فحص كيفية تطور المعرفة بمرور الوقت.
- **سلامة البيانات:** ضمان عدم فقدان البيانات عند التعطل بفضل الإطارات الثابتة.
---
## المفاهيم الأساسية
- **محرك الذاكرة الحية:** إضافة وتطوير الذاكرة باستمرار عبر الجلسات.
- **كبسولة السياق (`.mv2`):** كبسولات ذاكرة ذاتية الاحتواء وقابلة للمشاركة مع قواعد وصلاحية محددة.
- **تصحيح السفر عبر الزمن:** إرجاع أو إعادة تشغيل أو تفريع أي حالة من حالات الذاكرة.
- **الاستدعاء الذكي:** وصول محلي للذاكرة في أقل من 5 ملي ثانية مع ذاكرة تخزين مؤقت تنبؤية.
- **ذكاء الترميز:** يختار ويحدث تقنيات الضغط تلقائياً بمرور الوقت.
---
## حالات الاستخدام
نظرًا لأن Memvid يعمل دون اتصال بالإنترنت ومستقل عن النماذج، فإنه يُستخدم في:
- وكلاء الذكاء الاصطناعي طويلي الأمد.
- قواعد المعرفة للمؤسسات.
- أنظمة الذكاء الاصطناعي التي تعمل "بدون إنترنت أولاً".
- فهم الأكواد البرمجية (Codebases).
- المساعدين الشخصيين والأنظمة الطبية والقانونية والمالية.
---
## أدوات المطورين (SDKs)
| الحزمة | طريقة التثبيت |
| ----------------------------- | --------------------------- |
| **واجهة السطر البرمجي (CLI)** | `npm install -g memvid-cli` |
| **Node.js SDK** | `npm install @memvid/sdk` |
| **Python SDK** | `pip install memvid-sdk` |
| **Rust** | `cargo add memvid-core` |
---
## هيكل الملف
كل شيء يعيش داخل ملف واحد بصيغة `.mv2`:
```
┌────────────────────────────┐
│ Header (4KB) │ العنوان: النسخة والقدرة الاستيعابية
├────────────────────────────┤
│ Embedded WAL (1-64MB) │ سجل العمليات للتعافي من الأعطال
├────────────────────────────┤
│ Data Segments │ أجزاء البيانات: الإطارات المضغوطة
├────────────────────────────┤
│ Lex Index │ الفهرس اللغوي: البحث النصي الكامل
├────────────────────────────┤
│ Vec Index │ الفهرس المتجهي: البحث بالمتجهات (HNSW)
├────────────────────────────┤
│ Time Index │ الفهرس الزمني: الترتيب الزمني
├────────────────────────────┤
│ TOC (Footer) │ جدول المحتويات: مواقع الأجزاء
└────────────────────────────┘
```
---
## الدعم
هل لديك أسئلة؟
البريد الإلكتروني: contact@memvid.com
**لا تنسَ ترك ⭐ لدعم المشروع!**
---
## الترخيص
رخصة Apache 2.0 — راجع ملف [LICENSE](../../LICENSE) لمزيد من التفاصيل.
+379
View File
@@ -0,0 +1,379 @@
<!-- HEADER:START -->
<img width="2000" height="524" alt="Social Cover (9)" src="https://github.com/user-attachments/assets/cf66f045-c8be-494b-b696-b8d7e4fb709c" />
<!-- HEADER:END -->
<!-- FLAGS:START -->
<p align="center">
<a href="../../README.md">🇺🇸 English</a>
<a href="README.es.md">🇪🇸 Español</a>
<a href="README.fr.md">🇫🇷 Français</a>
<a href="README.so.md">🇸🇴 Soomaali</a>
<a href="README.ar.md">🇸🇦 العربية</a>
<a href="README.nl.md">🇧🇪/🇳🇱 Nederlands</a>
<a href="README.hi.md">🇮🇳 हिन्दी</a>
<a href="README.bn.md">🇧🇩 বাংলা</a>
<a href="README.cs.md">🇨🇿 Čeština</a>
<a href="README.ko.md">🇰🇷 한국어</a>
<a href="README.ja.md">🇯🇵 日本語</a>
<!-- Next Flag -->
</p>
<!-- FLAGS:END -->
<!-- NAV:START -->
<p align="center">
<a href="https://www.memvid.com">Website</a>
·
<a href="https://sandbox.memvid.com">Try Sandbox</a>
·
<a href="https://docs.memvid.com">Docs</a>
·
<a href="https://github.com/memvid/memvid/discussions">Discussions</a>
</p>
<!-- NAV:END -->
<!-- BADGES:START -->
<p align="center">
<a href="https://crates.io/crates/memvid-core"><img src="https://img.shields.io/crates/v/memvid-core?style=flat-square&logo=rust" alt="Crates.io" /></a>
<a href="https://docs.rs/memvid-core"><img src="https://img.shields.io/docsrs/memvid-core?style=flat-square&logo=docs.rs" alt="docs.rs" /></a>
<a href="https://github.com/memvid/memvid/blob/main/LICENSE"><img src="https://img.shields.io/badge/license-Apache%202.0-blue?style=flat-square" alt="License" /></a>
</p>
<p align="center">
<a href="https://github.com/memvid/memvid/stargazers"><img src="https://img.shields.io/github/stars/memvid/memvid?style=flat-square&logo=github" alt="Stars" /></a>
<a href="https://github.com/memvid/memvid/network/members"><img src="https://img.shields.io/github/forks/memvid/memvid?style=flat-square&logo=github" alt="Forks" /></a>
<a href="https://github.com/memvid/memvid/issues"><img src="https://img.shields.io/github/issues/memvid/memvid?style=flat-square&logo=github" alt="Issues" /></a>
<a href="https://discord.gg/2mynS7fcK7"><img src="https://img.shields.io/discord/1442910055233224745?style=flat-square&logo=discord&label=discord" alt="Discord" /></a>
</p>
<p align="center">
<a href="https://trendshift.io/repositories/17293" target="_blank"><img src="https://trendshift.io/api/badge/repositories/17293" alt="memvid%2Fmemvid | Trendshift" style="width: 250px; height: 55px;" width="250" height="55"/></a>
</p>
<!-- BADGES:END -->
# মেমভিড-এ অবদান (বাংলা অনুবাদ)
<p align="center">
<strong>মেমভিড হল এআই এজেন্টদের জন্য একটি একক-ফাইল মেমরি স্তর যার তাৎক্ষণিক পুনরুদ্ধার এবং দীর্ঘমেয়াদী মেমরি রয়েছে।</strong><br/>
ডাটাবেস ছাড়াই স্থায়ী, সংস্করণযুক্ত এবং পোর্টেবল মেমরি।
</p>
<p align="center">
<a href="https://www.memvid.com">ওয়েবসাইট</a>
·
<a href="https://sandbox.memvid.com">স্যান্ডবক্সি চেষ্টা করে দেখুন</a>
·
<a href="https://docs.memvid.com">ডক্স</a>
·
<a href="https://github.com/memvid/memvid/discussions">আলোচনা</a>
</p>
<p align="center">
<a href="https://crates.io/crates/memvid-core"><img src="https://img.shields.io/crates/v/memvid-core?style=flat-square&logo=rust" alt="Crates.io" /></a>
<a href="https://docs.rs/memvid-core"><img src="https://img.shields.io/docsrs/memvid-core?style=flat-square&logo=docs.rs" alt="docs.rs" /></a>
<a href="https://github.com/memvid/memvid/blob/main/LICENSE"><img src="https://img.shields.io/badge/license-Apache%202.0-blue?style=flat-square" alt="License" /></a>
</p>
<p align="center">
<a href="https://github.com/memvid/memvid/stargazers"><img src="https://img.shields.io/github/stars/memvid/memvid?style=flat-square&logo=github" alt="Stars" /></a>
<a href="https://github.com/memvid/memvid/network/members"><img src="https://img.shields.io/github/forks/memvid/memvid?style=flat-square&logo=github" alt="Forks" /></a>
<a href="https://github.com/memvid/memvid/issues"><img src="https://img.shields.io/github/issues/memvid/memvid?style=flat-square&logo=github" alt="Issues" /></a>
<a href="https://discord.gg/2mynS7fcK7"><img src="https://img.shields.io/discord/1442910055233224745?style=flat-square&logo=discord&label=discord" alt="Discord" /></a>
</p>
<p align="center">
<a href="https://trendshift.io/repositories/17293" target="_blank"><img src="https://trendshift.io/api/badge/repositories/17293" alt="memvid%2Fmemvid | Trendshift" style="width: 250px; height: 55px;" width="250" height="55"/</a>
</p>
<h2 align="center">⭐️ প্রকল্পটি সমর্থন করার জন্য একটি তারকা দিন। ⭐️</h2>
</p>
## মেমভিড কী?
মেমভিড হল একটি পোর্টেবল এআই মেমরি সিস্টেম যা আপনার ডেটা, এম্বেডিং, অনুসন্ধান কাঠামো এবং মেটাডেটা একটি একক ফাইলে প্যাক করে।
জটিল RAG পাইপলাইন বা সার্ভার-ভিত্তিক ভেক্টর ডাটাবেস চালানোর পরিবর্তে, Memvid ফাইল থেকে সরাসরি দ্রুত ডেটা পুনরুদ্ধারে সহায়তা করে।
ফলাফল হল একটি মডেল-অজ্ঞেয়বাদী, অবকাঠামো-মুক্ত মেমোরি স্তর যা AI এজেন্টদের স্থায়ী, দীর্ঘমেয়াদী মেমোরি দেয় যা তারা যেকোনো জায়গায় নিতে পারে।
---
## ভিডিও ফ্রেম কেন?
মেমভিড ভিডিও এনকোডিং থেকে অনুপ্রেরণা নেয়, ভিডিও সংরক্ষণের জন্য নয়, বরং এআই মেমোরিকে অ্যাপেন্ড-ওনলি, স্মার্ট ফ্রেমের অতি-দক্ষ সিকোয়েন্স হিসেবে সংগঠিত করার জন্য।
একটি স্মার্ট ফ্রেম হল একটি অ-পরিবর্তনযোগ্য ইউনিট যা টাইমস্ট্যাম্প, চেকসাম এবং মৌলিক মেটাডেটা সহ বিষয়বস্তু সংরক্ষণ করে।
ফ্রেমগুলিকে এমনভাবে গোষ্ঠীভুক্ত করা হয় যা দক্ষ কম্প্রেশন, ইনডেক্সিং এবং সমান্তরাল পঠনের সুযোগ করে দেয়।
এই ফ্রেম-ভিত্তিক নকশাটি সক্ষম করে:
- বিদ্যমান ডেটা পরিবর্তন বা দূষিত না করে কেবল ডেটা যোগ করা
- পূর্ববর্তী মেমরি অবস্থা অনুসন্ধান করা
- জ্ঞান কীভাবে বিকশিত হয় তার টাইমলাইন-স্টাইল তদন্ত
- প্রতিশ্রুতিবদ্ধ, অপরিবর্তনীয় ফ্রেমের মাধ্যমে ক্র্যাশ সুরক্ষা
- ভিডিও এনকোডিং থেকে গৃহীত কৌশল ব্যবহার করে দক্ষ কম্প্রেশন।
ফলাফল হল একটি একক ফাইল যা AI সিস্টেমের জন্য একটি রিওয়াইন্ডেবল মেমরি টাইমলাইন হিসাবে কাজ করে।
---
## মূল ধারণা
- **লিভিং মেমোরি ইঞ্জিন**
একটি সেশনের সময় স্থায়ী মেমোরি যোগ করুন, শাখা করুন এবং বিকশিত করুন।
- **ক্যাপসুল রেফারেন্স (`.mv2`)**
নিয়ম এবং মেয়াদোত্তীর্ণতা সহ স্বয়ংসম্পূর্ণ, শেয়ারযোগ্য মেমোরি ক্যাপসুল।
- **টাইম-ট্রাভেল ডিবাগিং**
যেকোন মেমোরি স্টেট রিওয়াইন্ড, রিপ্লে বা শাখা করুন।
- **স্মার্ট রিকল**
প্রেডিক্টিভ ক্যাশিং সহ সাব-5ms স্থানীয় মেমোরি অ্যাক্সেস।
- **কোডেক ইন্টেলিজেন্স**
সময়ের সাথে সাথে স্বয়ংক্রিয়ভাবে কম্প্রেশন নির্বাচন এবং আপগ্রেড করে।
---
## ব্যবহারের ক্ষেত্রে
মেমভিড হল একটি পোর্টেবল, সার্ভারলেস মেমরি স্তর যা এআই এজেন্টদের স্থায়ী মেমরি এবং দ্রুত রিকল দেয়। যেহেতু এটি মডেল-অ্যাগনস্টিক, মাল্টি-মডেল এবং সম্পূর্ণ অফলাইনে কাজ করে, তাই ডেভেলপাররা বিভিন্ন বাস্তব-বিশ্ব অ্যাপ্লিকেশনে মেমভিড ব্যবহার করছে।
- দীর্ঘমেয়াদী এআই এজেন্ট
- এন্টারপ্রাইজ জ্ঞান ভিত্তি
- অফলাইন-প্রথম এআই সিস্টেম
- কোডবেস বোঝা
- গ্রাহক সহায়তা এজেন্ট
- ওয়ার্কফ্লো অটোমেশন
- বিক্রয় এবং বিপণন সহ-পাইলট
- ব্যক্তিগত জ্ঞান সহকারী
- চিকিৎসা, আইনি এবং আর্থিক এজেন্ট
- নিরীক্ষণযোগ্য এবং ডিবাগযোগ্য এআই কর্মপ্রবাহ
- ​​কাস্টম অ্যাপ্লিকেশন
---
## SDKs & CLI
আপনার পছন্দের ভাষায় Memvid ব্যবহার করুন:
| Package | Install | Links |
| --------------- | --------------------------- | ------------------------------------------------------------------------------------------------------------------- |
| **CLI** | `npm install -g memvid-cli` | [![npm](https://img.shields.io/npm/v/memvid-cli?style=flat-square)](https://www.npmjs.com/package/memvid-cli) |
| **Node.js SDK** | `npm install @memvid/sdk` | [![npm](https://img.shields.io/npm/v/@memvid/sdk?style=flat-square)](https://www.npmjs.com/package/@memvid/sdk) |
| **Python SDK** | `pip install memvid-sdk` | [![PyPI](https://img.shields.io/pypi/v/memvid-sdk?style=flat-square)](https://pypi.org/project/memvid-sdk/) |
| **Rust** | `cargo add memvid-core` | [![Crates.io](https://img.shields.io/crates/v/memvid-core?style=flat-square)](https://crates.io/crates/memvid-core) |
---
## Installation (Rust)
### আবশ্যকতা
- **Rust 1.85.0+** — Install from [rustup.rs](https://rustup.rs)
### আপনার প্রকল্পে যোগ করুন
```toml
[dependencies]
memvid-core = "2.0"
```
### Feature Flags
| Feature | Description |
| ------------------- | ---------------------------------------------- |
| `lex` | Full-text search with BM25 ranking (Tantivy) |
| `pdf_extract` | Pure Rust PDF text extraction |
| `vec` | Vector similarity search (HNSW + ONNX) |
| `clip` | CLIP visual embeddings for image search |
| `whisper` | Audio transcription with Whisper |
| `temporal_track` | Natural language date parsing ("last Tuesday") |
| `parallel_segments` | Multi-threaded ingestion |
| `encryption` | Password-based encryption capsules (.mv2e) |
Enable features as needed:
```toml
[dependencies]
memvid-core = { version = "2.0", features = ["lex", "vec", "temporal_track"] }
```
---
## দ্রুত শুরু
```rust
use memvid_core::{Memvid, PutOptions, SearchRequest};
fn main() -> memvid_core::Result<()> {
// Create a new memory file
let mut mem = Memvid::create("knowledge.mv2")?;
// Add documents with metadata
let opts = PutOptions::builder()
.title("Meeting Notes")
.uri("mv2://meetings/2024-01-15")
.tag("project", "alpha")
.build();
mem.put_bytes_with_options(b"Q4 planning discussion...", opts)?;
mem.commit()?;
// Search
let response = mem.search(SearchRequest {
query: "planning".into(),
top_k: 10,
snippet_chars: 200,
..Default::default()
})?;
for hit in response.hits {
println!("{}: {}", hit.title.unwrap_or_default(), hit.text);
}
Ok(())
}
```
---
## তৈরি করুন
রিপোজিটরিটি ক্লোন করুন:
```bash
git clone https://github.com/memvid/memvid.git
cd memvid
```
ডিবাগ মোডে তৈরি করুন:
```bash
cargo build
```
রিলিজ মোডে তৈরি করুন (অপ্টিমাইজ করা):
```bash
cargo build --release
```
স্বতন্ত্র বৈশিষ্ট্য সহ তৈরি করুন:
```bash
cargo build --release --features "lex,vec,temporal_track"
```
---
## পরীক্ষাগুলি চালান
সকল পরীক্ষা চালান:
```bash
cargo test
```
নিম্নলিখিত আউটপুট দিয়ে পরীক্ষাটি চালান:
```bash
cargo test -- --nocapture
```
একটি নির্দিষ্ট পরীক্ষা চালান:
```bash
cargo test test_name
```
শুধুমাত্র ইন্টিগ্রেশন পরীক্ষা চালান:
```bash
cargo test --test lifecycle
cargo test --test search
cargo test --test mutation
```
---
## উদাহরণ
The `examples/` কার্যকরী ডিরেক্টরিগুলির উদাহরণ হল:
### মৌলিক ব্যবহার
এটি তৈরি, পুট, অনুসন্ধান এবং টাইমলাইন ক্রিয়াকলাপগুলি দেখায়:
```bash
cargo run --example basic_usage
```
### পিডিএফ ইনজেকশন
পিডিএফ ডকুমেন্টগুলি গ্রহণ করুন এবং অনুসন্ধান করুন ("Attention Is All You Need" পেপার ব্যবহার করে):
```bash
cargo run --example pdf_ingestion
```
### CLIP ভিজ্যুয়াল সার্চ
CLIP এম্বেডিং ব্যবহার করে ছবি সার্চ (`ক্লিপ` বৈশিষ্ট্য প্রয়োজন):
```bash
cargo run --example clip_visual_search --features clip
```
### হুইস্পার ট্রান্সক্রিপশন
অডিও ট্রান্সক্রিপশন (`হুইস্পার` বৈশিষ্ট্য প্রয়োজন):
```bash
cargo run --example test_whisper --features whisper
```
---
## ফাইল ফরম্যাট
সবকিছু একটি একক `.mv2` ফাইলের মধ্যে রয়েছে:
```
┌────────────────────────────┐
│ Header (4KB) │ Magic, version, capacity
├────────────────────────────┤
│ Embedded WAL (1-64MB) │ Crash recovery
├────────────────────────────┤
│ Data Segments │ Compressed frames
├────────────────────────────┤
│ Lex Index │ Tantivy full-text
├────────────────────────────┤
│ Vec Index │ HNSW vectors
├────────────────────────────┤
│ Time Index │ Chronological ordering
├────────────────────────────┤
│ TOC (Footer) │ Segment offsets
└────────────────────────────┘
```
কোন `.wal`, `.lock`, `.shm`, অথবা সাইডকার ফাইল নেই। কখনও না।
সম্পূর্ণ ফাইল ফর্ম্যাট স্পেসিফিকেশনের জন্য [MV2_SPEC.md](MV2_SPEC.md) দেখুন।
---
## সহায়তা
আপনার কি কোন প্রশ্ন বা প্রতিক্রিয়া আছে?
ইমেল: contact@memvid.com
**সমর্থন দেখানোর জন্য ⭐ দিন**
---
## লাইসেন্স
Apache License 2.0 — আরও তথ্যের জন্য [LICENSE](LICENSE) ফাইলটি দেখুন।
+424
View File
@@ -0,0 +1,424 @@
<!-- HEADER:START -->
<img width="2000" height="524" alt="Social Cover (9)" src="https://github.com/user-attachments/assets/cf66f045-c8be-494b-b696-b8d7e4fb709c" />
<!-- HEADER:END -->
<!-- FLAGS:START -->
<p align="center">
<a href="../../README.md">🇺🇸 English</a>
<a href="README.es.md">🇪🇸 Español</a>
<a href="README.fr.md">🇫🇷 Français</a>
<a href="README.so.md">🇸🇴 Soomaali</a>
<a href="README.ar.md">🇸🇦 العربية</a>
<a href="README.nl.md">🇧🇪/🇳🇱 Nederlands</a>
<a href="README.hi.md">🇮🇳 हिन्दी</a>
<a href="README.bn.md">🇧🇩 বাংলা</a>
<a href="README.cs.md">🇨🇿 Čeština</a>
<a href="README.ko.md">🇰🇷 한국어</a>
<a href="README.ja.md">🇯🇵 日本語</a>
<!-- Next Flag -->
</p>
<!-- FLAGS:END -->
<!-- NAV:START -->
<p align="center">
<a href="https://www.memvid.com">Website</a>
·
<a href="https://sandbox.memvid.com">Try Sandbox</a>
·
<a href="https://docs.memvid.com">Docs</a>
·
<a href="https://github.com/memvid/memvid/discussions">Discussions</a>
</p>
<!-- NAV:END -->
<!-- BADGES:START -->
<p align="center">
<a href="https://crates.io/crates/memvid-core"><img src="https://img.shields.io/crates/v/memvid-core?style=flat-square&logo=rust" alt="Crates.io" /></a>
<a href="https://docs.rs/memvid-core"><img src="https://img.shields.io/docsrs/memvid-core?style=flat-square&logo=docs.rs" alt="docs.rs" /></a>
<a href="https://github.com/memvid/memvid/blob/main/LICENSE"><img src="https://img.shields.io/badge/license-Apache%202.0-blue?style=flat-square" alt="License" /></a>
</p>
<p align="center">
<a href="https://github.com/memvid/memvid/stargazers"><img src="https://img.shields.io/github/stars/memvid/memvid?style=flat-square&logo=github" alt="Stars" /></a>
<a href="https://github.com/memvid/memvid/network/members"><img src="https://img.shields.io/github/forks/memvid/memvid?style=flat-square&logo=github" alt="Forks" /></a>
<a href="https://github.com/memvid/memvid/issues"><img src="https://img.shields.io/github/issues/memvid/memvid?style=flat-square&logo=github" alt="Issues" /></a>
<a href="https://discord.gg/2mynS7fcK7"><img src="https://img.shields.io/discord/1442910055233224745?style=flat-square&logo=discord&label=discord" alt="Discord" /></a>
</p>
<p align="center">
<a href="https://trendshift.io/repositories/17293" target="_blank"><img src="https://trendshift.io/api/badge/repositories/17293" alt="memvid%2Fmemvid | Trendshift" style="width: 250px; height: 55px;" width="250" height="55"/></a>
</p>
<!-- BADGES:END -->
<p align="center">
<strong>Memvid je jednosouborová paměťová vrstva pro AI agenty s okamžitým vyhledáváním a dlouhodobou pamětí.</strong><br/>
Trvalá, verzovaná a přenosná paměť, bez databází.
</p>
<h2 align="center">⭐️ Zanechte hvězdičku na podporu projektu ⭐️</h2>
</p>
## Co je Memvid?
Memvid je systém pro tvorbu AI pamětí, který balí vaše data, embeddingy, strukturu vyhledávání a metadata do jediného souboru.
Místo spouštění složitých RAG řešení nebo serverových vektorových databází umožňuje Memvid rychlé vyhledávání přímo ze souboru.
Výsledkem je modelově nezávislá paměťová vrstva bez infrastruktury, která poskytuje agentům AI trvalou, dlouhodobou paměť, kterou lze přenášet kamkoli.
---
## Co jsou inteligentní rámce?
Memvid čerpá inspiraci z enkódování videa, nikoli za účelem ukládání videa, ale za účelem **organizace paměti AI jako ultraefektivní sekvence inteligentních rámců, do kterých lze data pouze přidávat.**
Inteligentní rámec je neměnná jednotka, která ukládá obsah spolu s časovými značkami, kontrolními součty a základními metadaty.
Rámce jsou seskupeny tak, aby umožňovaly efektivní kompresi, indexování a paralelní čtení.
Tento design založený na rámcích umožňuje:
- Pouze zápisy bez úpravy nebo poškození existujících dat
- Dotazy na minulé stavy paměti
- Kontrolu vývoje znalostí ve stylu časové osy
- Bezpečnost proti selhání díky závazným, neměnným rámcům
- Efektivní kompresi pomocí technik převzatých z kódování videa
Výsledkem je jediný soubor, který se chová jako časová osa paměti pro systémy AI, ve které lze snadno hledat.
---
## Základní koncepty
- **Living Memory Engine**
Kontinuální přidávání, rozvětvování a vývoj paměti, napříč relacemi.
- **Capsule Context (`.mv2`)**
Samostatné, sdílené paměťové kapsle s pravidly a dobou platnosti.
- **Time-Travel Debugging**
Převíjení, přehrávání nebo rozvětvování libovolného stavu paměti.
- **Smart Recall**
Přístup k lokální paměti za méně než 5 ms s prediktivním ukládáním do mezipaměti.
- **Codec Intelligence**
Automaticky vybírá a vylepšuje kompresi v průběhu času.
---
## Případy použití
Memvid je přenosná paměťová vrstva bez serveru, která poskytuje agentům AI trvalou paměť a rychlé vyvolání. Protože je modelově nezávislá, multimodální a funguje zcela offline, vývojáři používají Memvid v široké škále reálných aplikací.
- Dlouhodobě běžící AI agenti
- Firemní znalostní báze
- Offline-First AI systémy
- Porozumění kódu
- Agenti zákaznické podpory
- Automatizace pracovních postupů
- Asistenti prodeje a marketingu
- Osobní znalostní asistenti
- Lékařští, právní a finanční agenti
- Auditovatelné a laditelné AI pracovní postupy
- Vlastní aplikace
---
## SDK a CLI
Používejte Memvid ve svém preferovaném jazyce:
| Balíček | Instalace | Odkazy |
| --------------- | --------------------------- | ------------------------------------------------------------------------------------------------------------------- |
| **CLI** | `npm install -g memvid-cli` | [![npm](https://img.shields.io/npm/v/memvid-cli?style=flat-square)](https://www.npmjs.com/package/memvid-cli) |
| **Node.js SDK** | `npm install @memvid/sdk` | [![npm](https://img.shields.io/npm/v/@memvid/sdk?style=flat-square)](https://www.npmjs.com/package/@memvid/sdk) |
| **Python SDK** | `pip install memvid-sdk` | [![PyPI](https://img.shields.io/pypi/v/memvid-sdk?style=flat-square)](https://pypi.org/project/memvid-sdk/) |
| **Rust** | `cargo add memvid-core` | [![Crates.io](https://img.shields.io/crates/v/memvid-core?style=flat-square)](https://crates.io/crates/memvid-core) |
---
## Instalace (Rust)
### Požadavky
- **Rust 1.85.0+** — Instalace z [rustup.rs](https://rustup.rs)
### Přidejte do svého projektu
```toml
[dependencies]
memvid-core = "2.0"
```
### Funkční příznaky
| Funkce | Popis |
| ------------------- | ---------------------------------------------------------- |
| `lex` | Fulltextové vyhledávání s hodnocením BM25 (Tantivy) |
| `pdf_extract` | Čistá extrakce textu z PDF v Rustu |
| `vec` | Vektorové vyhledávání podobnosti (HNSW + lokální vkládání textu přes ONNX) |
| `clip` | Vizuální vkládání CLIP pro vyhledávání obrázků |
| `whisper` | Přepis zvuku pomocí Whisper |
| `temporal_track` | Analýza datumu v přirozeném jazyce ("minulé úterý") |
| `parallel_segments` | Vícevláknové načítání |
| `encryption` | Kapsle šifrované pomocí hesla (.mv2e) |
Povolte funkce podle potřeby:
```toml
[dependencies]
memvid-core = { version = "2.0", features = ["lex", "vec", "temporal_track"] }
```
---
## Rychlý start
```rust
use memvid_core::{Memvid, PutOptions, SearchRequest};
fn main() -> memvid_core::Result<()> {
// Vytvoř nový paměťový soubor
let mut mem = Memvid::create("knowledge.mv2")?;
// Přidej dokumenty s metadaty
let opts = PutOptions::builder()
.title("Zápis jednání")
.uri("mv2://meetings/2024-01-15")
.tag("project", "alpha")
.build();
mem.put_bytes_with_options(b"Q4 plánované diskuze...", opts)?;
mem.commit()?;
// Vyhledávání
let response = mem.search(SearchRequest {
query: "planning".into(),
top_k: 10,
snippet_chars: 200,
..Default::default()
})?;
for hit in response.hits {
println!("{}: {}", hit.title.unwrap_or_default(), hit.text);
}
Ok(())
}
```
---
## Sestavení
Klonujte repozitář:
```bash
git clone https://github.com/memvid/memvid.git
cd memvid
```
Sestavení v režimu developkment:
```bash
cargo build
```
Sestavení v režimu production (optimalizované):
```bash
cargo build --release
```
Sestavení s konkrétními funkcemi:
```bash
cargo build --release --features "lex,vec,temporal_track"
```
---
## Spuštění testů
Spuštění všech testů:
```bash
cargo test
```
Spuštění testů s výstupem:
```bash
cargo test -- --nocapture
```
Spuštění konkrétního testu:
```bash
cargo test test_name
```
Spuštění pouze integračních testů:
```bash
cargo test --test lifecycle
cargo test --test search
cargo test --test mutation
```
---
## Příklady
Adresář `examples/` obsahuje funkční příklady:
### Základní použití
Demonstruje operace vytváření, vkládání, vyhledávání a práci s časovou osou:
```bash
cargo run --example basic_usage
```
### Načítání PDF
Načítání a vyhledávání v dokumentech PDF (demo používá dokument "Attention Is All You Need"):
```bash
cargo run --example pdf_ingestion
```
### Vizuální vyhledávání CLIP
Vyhledávání obrázků pomocí vložení CLIP (vyžaduje funkci `clip`):
```bash
cargo run --example clip_visual_search --features clip
```
### Přepis Whisper
Přepis zvuku (vyžaduje funkci `whisper`):
```bash
cargo run --example test_whisper --features whisper
```
---
## Modely vkládání textu
Funkce `vec` zahrnuje podporu lokálního vkládání textu pomocí modelů ONNX. Před použitím lokálního vkládání textu je nutné ručně stáhnout soubory modelů.
### Rychlý start: BGE-small (doporučeno)
Stáhněte si výchozí model BGE-small (384 dimenzí, rychlý a efektivní):
```bash
mkdir -p ~/.cache/memvid/text-models
# Download ONNX model
curl -L 'https://huggingface.co/BAAI/bge-small-en-v1.5/resolve/main/onnx/model.onnx' \
-o ~/.cache/memvid/text-models/bge-small-en-v1.5.onnx
# Download tokenizer
curl -L 'https://huggingface.co/BAAI/bge-small-en-v1.5/resolve/main/tokenizer.json' \
-o ~/.cache/memvid/text-models/bge-small-en-v1.5_tokenizer.json
```
### Dostupné modely
| Model | Rozměry | Velikost | Nejvhodnější pro |
| ----------------------- | ---------- | ------- | --------------------- |
| `bge-small-en-v1.5` | 384 | ~120 MB | Výchozí, rychlý |
| `bge-base-en-v1.5` | 768 | ~420 MB | Lepší kvalita |
| `nomic-embed-text-v1.5` | 768 | ~530 MB | Univerzální úkoly |
| `gte-large` | 1024 | ~1,3 GB | Nejvyšší kvalita |
### Další modely
**BGE-base** (768 dimensions):
```bash
curl -L 'https://huggingface.co/BAAI/bge-base-en-v1.5/resolve/main/onnx/model.onnx' \
-o ~/.cache/memvid/text-models/bge-base-en-v1.5.onnx
curl -L 'https://huggingface.co/BAAI/bge-base-en-v1.5/resolve/main/tokenizer.json' \
-o ~/.cache/memvid/text-models/bge-base-en-v1.5_tokenizer.json
```
**Nomic** (768 dimensions):
```bash
curl -L 'https://huggingface.co/nomic-ai/nomic-embed-text-v1.5/resolve/main/onnx/model.onnx' \
-o ~/.cache/memvid/text-models/nomic-embed-text-v1.5.onnx
curl -L 'https://huggingface.co/nomic-ai/nomic-embed-text-v1.5/resolve/main/tokenizer.json' \
-o ~/.cache/memvid/text-models/nomic-embed-text-v1.5_tokenizer.json
```
**GTE-large** (1024 dimensions):
```bash
curl -L 'https://huggingface.co/thenlper/gte-large/resolve/main/onnx/model.onnx' \
-o ~/.cache/memvid/text-models/gte-large.onnx
curl -L 'https://huggingface.co/thenlper/gte-large/resolve/main/tokenizer.json' \
-o ~/.cache/memvid/text-models/gte-large_tokenizer.json
```
### Použití v kódu
```rust
use memvid_core::text_embed::{LocalTextEmbedder, TextEmbedConfig};
use memvid_core::types::embedding::EmbeddingProvider;
// Použít výchozí model (BGE-small)
let config = TextEmbedConfig::default();
let embedder = LocalTextEmbedder::new(config)?;
let embedding = embedder.embed_text("hello world")?;
assert_eq!(embedding.len(), 384);
// Použijte jiný model
let config = TextEmbedConfig::bge_base();
let embedder = LocalTextEmbedder::new(config)?;
```
Kompletní příklad s výpočtem podobnosti a hodnocením vyhledávání najdete v souboru `examples/text_embedding.rs`.
---
## Formát souboru
Vše je uloženo v jediném souboru `.mv2`:
```
┌────────────────────────────┐
│ Záhlaví (4 KB) │ Magie, verze, kapacita
├────────────────────────────┤
│ Embedded WAL (1-64 MB) │ Obnova po selhání
├────────────────────────────┤
│ Datové segmenty │ Komprimované rámce
├────────────────────────────┤
│ Lex Index │ Tantivy full-text
├────────────────────────────┤
│ Vec Index │ Vektory HNSW
├────────────────────────────┤
│ Time Index │ Chronologické řazení
├────────────────────────────┤
│ TOC (zápatí) │ Segmentové posuny
└────────────────────────────┘
```
Žádné další soubory, jako je `.wal`, `.lock`, `.shm` nejsou potřeba. Nikdy.
Kompletní specifikace formátu souboru najdete v [MV2_SPEC.md](MV2_SPEC.md).
---
## Podpora
Máte dotazy nebo připomínky?
E-mail: contact@memvid.com
**Dejte ⭐ a projevte svou podporu**
---
## Licence
Apache License 2.0 — podrobnosti najdete v souboru [LICENSE](LICENSE).
+500
View File
@@ -0,0 +1,500 @@
<!-- HEADER:START -->
<img width="2000" height="524" alt="Social Cover (9)" src="https://github.com/user-attachments/assets/cf66f045-c8be-494b-b696-b8d7e4fb709c" />
<!-- HEADER:END -->
<!-- FLAGS:START -->
<p align="center">
<a href="../../README.md">🇺🇸 English</a>
<a href="README.es.md">🇪🇸 Español</a>
<a href="README.fr.md">🇫🇷 Français</a>
<a href="README.so.md">🇸🇴 Soomaali</a>
<a href="README.ar.md">🇸🇦 العربية</a>
<a href="README.nl.md">🇧🇪/🇳🇱 Nederlands</a>
<a href="README.hi.md">🇮🇳 हिन्दी</a>
<a href="README.bn.md">🇧🇩 বাংলা</a>
<a href="README.cs.md">🇨🇿 Čeština</a>
<a href="README.ko.md">🇰🇷 한국어</a>
<a href="README.ja.md">🇯🇵 日本語</a>
<!-- Next Flag -->
</p>
<!-- FLAGS:END -->
<!-- NAV:START -->
<p align="center">
<a href="https://www.memvid.com">Website</a>
·
<a href="https://sandbox.memvid.com">Try Sandbox</a>
·
<a href="https://docs.memvid.com">Docs</a>
·
<a href="https://github.com/memvid/memvid/discussions">Discussions</a>
</p>
<!-- NAV:END -->
<!-- BADGES:START -->
<p align="center">
<a href="https://crates.io/crates/memvid-core"><img src="https://img.shields.io/crates/v/memvid-core?style=flat-square&logo=rust" alt="Crates.io" /></a>
<a href="https://docs.rs/memvid-core"><img src="https://img.shields.io/docsrs/memvid-core?style=flat-square&logo=docs.rs" alt="docs.rs" /></a>
<a href="https://github.com/memvid/memvid/blob/main/LICENSE"><img src="https://img.shields.io/badge/license-Apache%202.0-blue?style=flat-square" alt="License" /></a>
</p>
<p align="center">
<a href="https://github.com/memvid/memvid/stargazers"><img src="https://img.shields.io/github/stars/memvid/memvid?style=flat-square&logo=github" alt="Stars" /></a>
<a href="https://github.com/memvid/memvid/network/members"><img src="https://img.shields.io/github/forks/memvid/memvid?style=flat-square&logo=github" alt="Forks" /></a>
<a href="https://github.com/memvid/memvid/issues"><img src="https://img.shields.io/github/issues/memvid/memvid?style=flat-square&logo=github" alt="Issues" /></a>
<a href="https://discord.gg/2mynS7fcK7"><img src="https://img.shields.io/discord/1442910055233224745?style=flat-square&logo=discord&label=discord" alt="Discord" /></a>
</p>
<p align="center">
<a href="https://trendshift.io/repositories/17293" target="_blank"><img src="https://trendshift.io/api/badge/repositories/17293" alt="memvid%2Fmemvid | Trendshift" style="width: 250px; height: 55px;" width="250" height="55"/></a>
</p>
<!-- BADGES:END -->
<p align="center">
<strong>Memvid es una capa de memoria de un solo archivo para agentes de IA, con recuperación instantánea y memoria a largo plazo.</strong><br/>
Memoria persistente, versionada y portable, sin bases de datos.
</p>
<h2 align="center">⭐️ Deja una STAR para apoyar el proyecto ⭐️</h2>
</p>
## Lo más destacado de los benchmarks
**🚀 Mayor precisión que cualquier otro sistema de memoria:** +35% SOTA en LoCoMo, con recall y razonamiento conversacional de largo horizonte de primer nivel.
**🧠 Mejor razonamiento multi-hop y temporal:** +76% en multi-hop y +56% en temporal frente al promedio de la industria.
**⚡ Latencia ultra baja a escala:** 0.025 ms P50 y 0.075 ms P99, con 1,372× más throughput que los enfoques estándar.
**🔬 Benchmarks totalmente reproducibles:** LoCoMo (10 conversaciones de ~26K tokens), evaluación open source y LLM-as-Judge.
---
## ¿Qué es Memvid?
Memvid es un sistema de memoria portable para IA que empaqueta tus datos, embeddings, estructura de búsqueda y metadatos en un solo archivo.
En lugar de ejecutar pipelines RAG complejos o bases de datos vectoriales basadas en servidor, Memvid permite una recuperación rápida directamente desde el archivo.
El resultado es una capa de memoria agnóstica al modelo, sin infraestructura, que da a los agentes de IA una memoria persistente y a largo plazo que pueden llevar a cualquier parte.
---
## ¿Por qué fotogramas de vídeo?
Memvid se inspira en la codificación de vídeo, no para almacenar vídeo, sino para **organizar la memoria de IA como una secuencia de Smart Frames ultrarrápida y append-only.**
Un Smart Frame es una unidad inmutable que almacena contenido junto con marcas de tiempo (timestamps), checksums y metadatos básicos.
Los frames se agrupan de una forma que permite una compresión, indexación y lecturas paralelas eficientes.
Este diseño basado en frames permite:
- Escrituras append-only sin modificar ni corromper los datos existentes
- Consultas sobre estados pasados de la memoria
- Inspección estilo línea temporal (timeline) de cómo evoluciona el conocimiento
- Seguridad ante fallos (crash safety) mediante frames confirmados e inmutables
- Compresión eficiente usando técnicas adaptadas de la codificación de vídeo
El resultado es un único archivo que se comporta como una línea temporal de memoria “rebobinable” para sistemas de IA.
---
## Conceptos principales
- **Living Memory Engine**
Añade, ramifica (branch) y evoluciona la memoria de forma continua entre sesiones.
- **Capsule Context (`.mv2`)**
Cápsulas de memoria autocontenidas y compartibles, con reglas y caducidad.
- **Time-Travel Debugging**
Rebobina, reproduce (replay) o ramifica cualquier estado de memoria.
- **Smart Recall**
Acceso local a memoria en menos de 5ms con caché predictiva.
- **Codec Intelligence**
Selecciona y actualiza la compresión automáticamente con el tiempo.
---
## Casos de uso
Memvid es una capa de memoria portable y serverless que da a los agentes de IA memoria persistente y recuerdo rápido. Como es agnóstica al modelo, multi-modal y funciona totalmente offline, los desarrolladores están usando Memvid en una amplia gama de aplicaciones reales.
- Agentes de IA de larga duración
- Bases de conocimiento empresariales
- Sistemas de IA offline-first
- Comprensión de codebases
- Agentes de soporte al cliente
- Automatización de flujos de trabajo
- Copilotos de ventas y marketing
- Asistentes de conocimiento personal
- Agentes médicos, legales y financieros
- Flujos de trabajo de IA auditables y depurables
- Aplicaciones personalizadas
---
## SDKs & CLI
Usa Memvid en tu lenguaje preferido:
| Package | Install | Links |
| --------------- | --------------------------- | ------------------------------------------------------------------------------------------------------------------- |
| **CLI** | `npm install -g memvid-cli` | [![npm](https://img.shields.io/npm/v/memvid-cli?style=flat-square)](https://www.npmjs.com/package/memvid-cli) |
| **Node.js SDK** | `npm install @memvid/sdk` | [![npm](https://img.shields.io/npm/v/@memvid/sdk?style=flat-square)](https://www.npmjs.com/package/@memvid/sdk) |
| **Python SDK** | `pip install memvid-sdk` | [![PyPI](https://img.shields.io/pypi/v/memvid-sdk?style=flat-square)](https://pypi.org/project/memvid-sdk/) |
| **Rust** | `cargo add memvid-core` | [![Crates.io](https://img.shields.io/crates/v/memvid-core?style=flat-square)](https://crates.io/crates/memvid-core) |
---
## Instalación (Rust)
### Requisitos
- **Rust 1.85.0+** — Instálalo desde [rustup.rs](https://rustup.rs)
### Añadir a tu proyecto
```toml
[dependencies]
memvid-core = "2.0"
```
### Feature Flags
| Feature | Descripción |
| ------------------- | ------------------------------------------------------------------ |
| `lex` | Búsqueda full-text con ranking BM25 (Tantivy) |
| `pdf_extract` | Extracción de texto PDF 100% en Rust |
| `vec` | Búsqueda por similitud vectorial (HNSW + embeddings locales vía ONNX) |
| `clip` | Embeddings visuales CLIP para búsqueda de imágenes |
| `whisper` | Transcripción de audio con Whisper |
| `api_embed` | Embeddings en la nube mediante API (OpenAI) |
| `temporal_track` | Interpretación de fechas en lenguaje natural ("el martes pasado") |
| `parallel_segments` | Ingesta multi-hilo |
| `encryption` | Cápsulas cifradas con contraseña (.mv2e) |
| `symspell_cleanup` | Reparación robusta de texto PDF (corrige "emp lo yee" -> "employee") |
Activa las features según lo necesites:
```toml
[dependencies]
memvid-core = { version = "2.0", features = ["lex", "vec", "temporal_track"] }
```
---
## Inicio rápido
```rust
use memvid_core::{Memvid, PutOptions, SearchRequest};
fn main() -> memvid_core::Result<()> {
// Create a new memory file
let mut mem = Memvid::create("knowledge.mv2")?;
// Add documents with metadata
let opts = PutOptions::builder()
.title("Meeting Notes")
.uri("mv2://meetings/2024-01-15")
.tag("project", "alpha")
.build();
mem.put_bytes_with_options(b"Q4 planning discussion...", opts)?;
mem.commit()?;
// Search
let response = mem.search(SearchRequest {
query: "planning".into(),
top_k: 10,
snippet_chars: 200,
..Default::default()
})?;
for hit in response.hits {
println!("{}: {}", hit.title.unwrap_or_default(), hit.text);
}
Ok(())
}
```
---
## Build
Clona el repositorio:
```bash
git clone https://github.com/memvid/memvid.git
cd memvid
```
Compila en modo debug:
```bash
cargo build
```
Compila en modo release (optimizado):
```bash
cargo build --release
```
Compila con features específicas:
```bash
cargo build --release --features "lex,vec,temporal_track"
```
---
## Ejecutar tests
Ejecuta todos los tests:
```bash
cargo test
```
Ejecuta los tests con salida:
```bash
cargo test -- --nocapture
```
Ejecuta un test específico:
```bash
cargo test test_name
```
Ejecuta solo tests de integración:
```bash
cargo test --test lifecycle
cargo test --test search
cargo test --test mutation
```
---
## Ejemplos
El directorio `examples/` contiene ejemplos funcionales:
### Uso básico
Demuestra operaciones de create, put, search y timeline:
```bash
cargo run --example basic_usage
```
### Ingesta de PDF
Ingiere y busca documentos PDF (usa el paper “Attention Is All You Need”):
```bash
cargo run --example pdf_ingestion
```
### Búsqueda visual con CLIP
Búsqueda de imágenes usando embeddings de CLIP (requiere la feature `clip`):
```bash
cargo run --example clip_visual_search --features clip
```
### Transcripción con Whisper
Transcripción de audio (requiere la feature `whisper`):
```bash
cargo run --example test_whisper --features whisper
```
---
## Modelos de embeddings de texto
La feature `vec` incluye soporte para embeddings de texto locales usando modelos ONNX. Antes de usar embeddings locales, necesitas descargar manualmente los archivos del modelo.
### Inicio rápido: BGE-small (recomendado)
Descarga el modelo BGE-small por defecto (384 dimensiones, rápido y eficiente):
```bash
mkdir -p ~/.cache/memvid/text-models
# Descargar modelo ONNX
curl -L 'https://huggingface.co/BAAI/bge-small-en-v1.5/resolve/main/onnx/model.onnx' \
-o ~/.cache/memvid/text-models/bge-small-en-v1.5.onnx
# Descargar tokenizer
curl -L 'https://huggingface.co/BAAI/bge-small-en-v1.5/resolve/main/tokenizer.json' \
-o ~/.cache/memvid/text-models/bge-small-en-v1.5_tokenizer.json
```
### Modelos disponibles
| Modelo | Dimensiones | Tamaño | Mejor para |
| ----------------------- | ----------- | ------ | --------------------- |
| `bge-small-en-v1.5` | 384 | ~120MB | Opción por defecto, rápido |
| `bge-base-en-v1.5` | 768 | ~420MB | Mejor calidad |
| `nomic-embed-text-v1.5` | 768 | ~530MB | Tareas versátiles |
| `gte-large` | 1024 | ~1.3GB | Máxima calidad |
### Otros modelos
**BGE-base** (768 dimensiones):
```bash
curl -L 'https://huggingface.co/BAAI/bge-base-en-v1.5/resolve/main/onnx/model.onnx' \
-o ~/.cache/memvid/text-models/bge-base-en-v1.5.onnx
curl -L 'https://huggingface.co/BAAI/bge-base-en-v1.5/resolve/main/tokenizer.json' \
-o ~/.cache/memvid/text-models/bge-base-en-v1.5_tokenizer.json
```
**Nomic** (768 dimensiones):
```bash
curl -L 'https://huggingface.co/nomic-ai/nomic-embed-text-v1.5/resolve/main/onnx/model.onnx' \
-o ~/.cache/memvid/text-models/nomic-embed-text-v1.5.onnx
curl -L 'https://huggingface.co/nomic-ai/nomic-embed-text-v1.5/resolve/main/tokenizer.json' \
-o ~/.cache/memvid/text-models/nomic-embed-text-v1.5_tokenizer.json
```
**GTE-large** (1024 dimensiones):
```bash
curl -L 'https://huggingface.co/thenlper/gte-large/resolve/main/onnx/model.onnx' \
-o ~/.cache/memvid/text-models/gte-large.onnx
curl -L 'https://huggingface.co/thenlper/gte-large/resolve/main/tokenizer.json' \
-o ~/.cache/memvid/text-models/gte-large_tokenizer.json
```
### Uso en código
```rust
use memvid_core::text_embed::{LocalTextEmbedder, TextEmbedConfig};
use memvid_core::types::embedding::EmbeddingProvider;
// Usar el modelo por defecto (BGE-small)
let config = TextEmbedConfig::default();
let embedder = LocalTextEmbedder::new(config)?;
let embedding = embedder.embed_text("hello world")?;
assert_eq!(embedding.len(), 384);
// Usar un modelo distinto
let config = TextEmbedConfig::bge_base();
let embedder = LocalTextEmbedder::new(config)?;
```
Consulta `examples/text_embedding.rs` para ver un ejemplo completo con cálculo de similitud y ranking de búsqueda.
### Consistencia del modelo
Para evitar mezclar modelos por accidente, por ejemplo consultar un índice BGE-small con embeddings de OpenAI, puedes asociar explícitamente tu instancia de Memvid a un nombre de modelo:
```rust
// Vincula el índice a un modelo concreto.
// Si el índice ya fue creado con otro modelo, devolverá un error.
mem.set_vec_model("bge-small-en-v1.5")?;
```
Esta vinculación es persistente. Una vez definida, cualquier intento futuro de usar otro nombre de modelo fallará de inmediato con un error `ModelMismatch`.
---
## Embeddings por API (OpenAI)
La feature `api_embed` habilita la generación de embeddings en la nube usando la API de OpenAI.
### Configuración
Define tu clave de API de OpenAI:
```bash
export OPENAI_API_KEY="sk-..."
```
### Uso
```rust
use memvid_core::api_embed::{OpenAIConfig, OpenAIEmbedder};
use memvid_core::types::embedding::EmbeddingProvider;
// Usar el modelo por defecto (text-embedding-3-small)
let config = OpenAIConfig::default();
let embedder = OpenAIEmbedder::new(config)?;
let embedding = embedder.embed_text("hello world")?;
assert_eq!(embedding.len(), 1536);
// Usar un modelo de mayor calidad
let config = OpenAIConfig::large(); // text-embedding-3-large (3072 dims)
let embedder = OpenAIEmbedder::new(config)?;
```
### Modelos disponibles
| Modelo | Dimensiones | Mejor para |
| ------------------------ | ----------- | -------------------------------- |
| `text-embedding-3-small` | 1536 | Por defecto, más rápido y económico |
| `text-embedding-3-large` | 3072 | Máxima calidad |
| `text-embedding-ada-002` | 1536 | Modelo heredado |
Consulta `examples/openai_embedding.rs` para ver un ejemplo completo.
---
## Formato de archivo
Todo vive en un único archivo `.mv2`:
```
┌────────────────────────────┐
│ Header (4KB) │ Magic, version, capacity
├────────────────────────────┤
│ Embedded WAL (1-64MB) │ Crash recovery
├────────────────────────────┤
│ Data Segments │ Compressed frames
├────────────────────────────┤
│ Lex Index │ Tantivy full-text
├────────────────────────────┤
│ Vec Index │ HNSW vectors
├────────────────────────────┤
│ Time Index │ Chronological ordering
├────────────────────────────┤
│ TOC (Footer) │ Segment offsets
└────────────────────────────┘
```
Sin archivos `.wal`, `.lock`, `.shm` ni sidecars. Nunca.
Consulta [MV2_SPEC.md](MV2_SPEC.md) para la especificación completa del formato de archivo.
---
## Soporte
¿Tienes preguntas o feedback?
Email: contact@memvid.com
**Deja una ⭐ para mostrar apoyo**
---
> **Memvid v1 (memoria basada en QR) está obsoleto**
>
> Si estás viendo referencias a códigos QR, estás usando información desactualizada.
>
> Consulta: https://docs.memvid.com/memvid-v1-deprecation
---
## Licencia
Apache License 2.0 — consulta el archivo [LICENSE](LICENSE) para más detalles.
+346
View File
@@ -0,0 +1,346 @@
<!-- HEADER:START -->
<img width="2000" height="524" alt="Social Cover (9)" src="https://github.com/user-attachments/assets/cf66f045-c8be-494b-b696-b8d7e4fb709c" />
<!-- HEADER:END -->
<!-- FLAGS:START -->
<p align="center">
<a href="../../README.md">🇺🇸 English</a>
<a href="README.es.md">🇪🇸 Español</a>
<a href="README.fr.md">🇫🇷 Français</a>
<a href="README.so.md">🇸🇴 Soomaali</a>
<a href="README.ar.md">🇸🇦 العربية</a>
<a href="README.nl.md">🇧🇪/🇳🇱 Nederlands</a>
<a href="README.hi.md">🇮🇳 हिन्दी</a>
<a href="README.bn.md">🇧🇩 বাংলা</a>
<a href="README.cs.md">🇨🇿 Čeština</a>
<a href="README.ko.md">🇰🇷 한국어</a>
<a href="README.ja.md">🇯🇵 日本語</a>
<!-- Next Flag -->
</p>
<!-- FLAGS:END -->
<!-- NAV:START -->
<p align="center">
<a href="https://www.memvid.com">Website</a>
·
<a href="https://sandbox.memvid.com">Try Sandbox</a>
·
<a href="https://docs.memvid.com">Docs</a>
·
<a href="https://github.com/memvid/memvid/discussions">Discussions</a>
</p>
<!-- NAV:END -->
<!-- BADGES:START -->
<p align="center">
<a href="https://crates.io/crates/memvid-core"><img src="https://img.shields.io/crates/v/memvid-core?style=flat-square&logo=rust" alt="Crates.io" /></a>
<a href="https://docs.rs/memvid-core"><img src="https://img.shields.io/docsrs/memvid-core?style=flat-square&logo=docs.rs" alt="docs.rs" /></a>
<a href="https://github.com/memvid/memvid/blob/main/LICENSE"><img src="https://img.shields.io/badge/license-Apache%202.0-blue?style=flat-square" alt="License" /></a>
</p>
<p align="center">
<a href="https://github.com/memvid/memvid/stargazers"><img src="https://img.shields.io/github/stars/memvid/memvid?style=flat-square&logo=github" alt="Stars" /></a>
<a href="https://github.com/memvid/memvid/network/members"><img src="https://img.shields.io/github/forks/memvid/memvid?style=flat-square&logo=github" alt="Forks" /></a>
<a href="https://github.com/memvid/memvid/issues"><img src="https://img.shields.io/github/issues/memvid/memvid?style=flat-square&logo=github" alt="Issues" /></a>
<a href="https://discord.gg/2mynS7fcK7"><img src="https://img.shields.io/discord/1442910055233224745?style=flat-square&logo=discord&label=discord" alt="Discord" /></a>
</p>
<p align="center">
<a href="https://trendshift.io/repositories/17293" target="_blank"><img src="https://trendshift.io/api/badge/repositories/17293" alt="memvid%2Fmemvid | Trendshift" style="width: 250px; height: 55px;" width="250" height="55"/></a>
</p>
<!-- BADGES:END -->
<p align="center">
<strong>Memvid est une couche mémoire à fichier unique pour agents IA, avec récupération instantanée et mémoire long terme.</strong><br/>
Mémoire persistante, versionnée et portable, sans bases de données.
</p>
<h2 align="center">⭐️ Laissez une STAR pour soutenir le projet ⭐️</h2>
</p>
## Qu'est-ce que Memvid ?
Memvid est un système de mémoire IA portable qui regroupe vos données, embeddings, structure de recherche et métadonnées dans un seul fichier.
Au lieu d'exécuter des pipelines RAG complexes ou des bases de données vectorielles côté serveur, Memvid permet une récupération rapide directement depuis le fichier.
Le résultat est une couche mémoire agnostique au modèle, sans infrastructure, qui donne aux agents IA une mémoire persistante et longue durée qu'ils peuvent emporter partout.
---
## Pourquoi des images vidéo ?
Memvid s'inspire de l'encodage vidéo, non pas pour stocker de la vidéo, mais pour **organiser la mémoire IA en une séquence append-only ultra-efficace de Smart Frames.**
Une Smart Frame est une unité immuable qui stocke le contenu avec des horodatages, des checksums et des métadonnées de base.
Les frames sont regroupées d'une manière qui permet une compression, une indexation et des lectures parallèles efficaces.
Ce design basé sur les frames permet :
- Écritures append-only sans modifier ni corrompre les données existantes
- Requêtes sur des états mémoire passés
- Inspection type timeline de l'évolution des connaissances
- Sécurité en cas de crash via des frames immuables et validées
- Compression efficace grâce à des techniques adaptées de l'encodage vidéo
Le résultat est un fichier unique qui se comporte comme une timeline mémoire rembobinable pour les systèmes IA.
---
## Concepts de base
- **Moteur de mémoire vivant**
Ajoutez, branchez et faites évoluer la mémoire en continu entre les sessions.
- **Capsule de Contexte (`.mv2`)**
Capsules mémoire autonomes et partageables avec règles et expiration.
- **Débogage par 'voyage temporel'**
Rembobinez, rejouez ou branchez n'importe quel état mémoire.
- **Rappel intelligent**
Accès mémoire local en moins de 5 ms avec cache prédictif.
- **Intelligence du codec**
Sélection et mise à niveau automatiques de la compression au fil du temps.
---
## Cas d'usage
Memvid est une couche mémoire portable et sans serveur qui donne aux agents IA une mémoire persistante et un rappel rapide. Parce qu'il est agnostique au modèle, multimodal et fonctionne entièrement hors ligne, les développeurs utilisent Memvid pour un large éventail d'applications réelles.
- Agents IA longue durée
- Bases de connaissances d'entreprise
- Systèmes IA offline-first
- Compréhension de codebase
- Agents de support client
- Automatisation des workflows
- Copilotes ventes et marketing
- Assistants de connaissance personnels
- Agents médicaux, juridiques et financiers
- Workflows IA auditables et débogables
- Applications sur mesure
---
## SDKs & CLI
Utilisez Memvid dans votre langage préféré :
| Package | Installation | Liens |
|---------|---------|-------|
| **CLI** | `npm install -g memvid-cli` | [![npm](https://img.shields.io/npm/v/memvid-cli?style=flat-square)](https://www.npmjs.com/package/memvid-cli) |
| **Node.js SDK** | `npm install @memvid/sdk` | [![npm](https://img.shields.io/npm/v/@memvid/sdk?style=flat-square)](https://www.npmjs.com/package/@memvid/sdk) |
| **Python SDK** | `pip install memvid-sdk` | [![PyPI](https://img.shields.io/pypi/v/memvid-sdk?style=flat-square)](https://pypi.org/project/memvid-sdk/) |
| **Rust** | `cargo add memvid-core` | [![Crates.io](https://img.shields.io/crates/v/memvid-core?style=flat-square)](https://crates.io/crates/memvid-core) |
---
## Installation (Rust)
### Prérequis
- **Rust 1.85.0+** — Installer depuis [rustup.rs](https://rustup.rs)
### Ajouter à votre projet
```toml
[dependencies]
memvid-core = "2.0"
```
### Feature Flags
| Feature | Description |
|---------|-------------|
| `lex` | Full-text search with BM25 ranking (Tantivy) |
| `pdf_extract` | Pure Rust PDF text extraction |
| `vec` | Vector similarity search (HNSW + ONNX) |
| `clip` | CLIP visual embeddings for image search |
| `whisper` | Audio transcription with Whisper |
| `temporal_track` | Natural language date parsing ("last Tuesday") |
| `parallel_segments` | Multi-threaded ingestion |
| `encryption` | Password-based encryption capsules (.mv2e) |
Activez les features selon vos besoins :
```toml
[dependencies]
memvid-core = { version = "2.0", features = ["lex", "vec", "temporal_track"] }
```
---
## Démarrage rapide
```rust
use memvid_core::{Memvid, PutOptions, SearchRequest};
fn main() -> memvid_core::Result<()> {
// Créer un nouveau fichier de mémoire
let mut mem = Memvid::create("knowledge.mv2")?;
// Ajouter des documents avec des métadonnées
let opts = PutOptions::builder()
.title("Meeting Notes")
.uri("mv2://meetings/2024-01-15")
.tag("project", "alpha")
.build();
mem.put_bytes_with_options(b"Q4 planning discussion...", opts)?;
mem.commit()?;
// Rechercher
let response = mem.search(SearchRequest {
query: "planning".into(),
top_k: 10,
snippet_chars: 200,
..Default::default()
})?;
for hit in response.hits {
println!("{}: {}", hit.title.unwrap_or_default(), hit.text);
}
Ok(())
}
```
---
## Compiler
Cloner le repository :
```bash
git clone https://github.com/memvid/memvid.git
cd memvid
```
Compiler en mode debug :
```bash
cargo build
```
Compiler en mode release (optimisé) :
```bash
cargo build --release
```
Compiler avec des features spécifiques :
```bash
cargo build --release --features "lex,vec,temporal_track"
```
---
## Exécuter les tests
Exécuter tous les tests :
```bash
cargo test
```
Exécuter les tests avec sortie :
```bash
cargo test -- --nocapture
```
Exécuter un test spécifique :
```bash
cargo test test_name
```
Exécuter uniquement les tests d'intégration :
```bash
cargo test --test lifecycle
cargo test --test search
cargo test --test mutation
```
---
## Exemples
Le répertoire `examples/` contient des exemples fonctionnels :
### Utilisation de base
Démontre create, put, search et les opérations de timeline :
```bash
cargo run --example basic_usage
```
### Ingestion de PDF
Ingérer et rechercher des documents PDF (utilise l'article "Attention Is All You Need") :
```bash
cargo run --example pdf_ingestion
```
### Recherche visuelle CLIP
Recherche d'images à l'aide d'embeddings CLIP (nécessite la feature `clip`) :
```bash
cargo run --example clip_visual_search --features clip
```
### Transcription Whisper
Transcription audio (nécessite la feature `whisper`) :
```bash
cargo run --example test_whisper --features whisper
```
---
## Format de fichier
Tout est dans un seul fichier `.mv2` :
```
┌────────────────────────────┐
│ Header (4KB) │ Magic, version, capacity
├────────────────────────────┤
│ Embedded WAL (1-64MB) │ Crash recovery
├────────────────────────────┤
│ Data Segments │ Compressed frames
├────────────────────────────┤
│ Lex Index │ Tantivy full-text
├────────────────────────────┤
│ Vec Index │ HNSW vectors
├────────────────────────────┤
│ Time Index │ Chronological ordering
├────────────────────────────┤
│ TOC (Footer) │ Segment offsets
└────────────────────────────┘
```
Pas de `.wal`, `.lock`, `.shm` ou fichiers auxiliaires. Jamais.
Voir [MV2_SPEC.md](MV2_SPEC.md) pour la spécification complète du format de fichier.
---
## Support
Vous avez des questions ou des retours ?
Email : contact@memvid.com
**Laissez une ⭐ pour montrer votre soutien**
---
## Licence
Apache License 2.0 — voir le fichier [LICENSE](LICENSE) pour plus de détails.
+347
View File
@@ -0,0 +1,347 @@
<!-- HEADER:START -->
<img width="2000" height="524" alt="Social Cover (9)" src="https://github.com/user-attachments/assets/cf66f045-c8be-494b-b696-b8d7e4fb709c" />
<!-- HEADER:END -->
<!-- FLAGS:START -->
<p align="center">
<a href="../../README.md">🇺🇸 English</a>
<a href="README.es.md">🇪🇸 Español</a>
<a href="README.fr.md">🇫🇷 Français</a>
<a href="README.so.md">🇸🇴 Soomaali</a>
<a href="README.ar.md">🇸🇦 العربية</a>
<a href="README.nl.md">🇧🇪/🇳🇱 Nederlands</a>
<a href="README.hi.md">🇮🇳 हिन्दी</a>
<a href="README.bn.md">🇧🇩 বাংলা</a>
<a href="README.cs.md">🇨🇿 Čeština</a>
<a href="README.ko.md">🇰🇷 한국어</a>
<a href="README.ja.md">🇯🇵 日本語</a>
<!-- Next Flag -->
</p>
<!-- FLAGS:END -->
<!-- NAV:START -->
<p align="center">
<a href="https://www.memvid.com">Website</a>
·
<a href="https://sandbox.memvid.com">Try Sandbox</a>
·
<a href="https://docs.memvid.com">Docs</a>
·
<a href="https://github.com/memvid/memvid/discussions">Discussions</a>
</p>
<!-- NAV:END -->
<!-- BADGES:START -->
<p align="center">
<a href="https://crates.io/crates/memvid-core"><img src="https://img.shields.io/crates/v/memvid-core?style=flat-square&logo=rust" alt="Crates.io" /></a>
<a href="https://docs.rs/memvid-core"><img src="https://img.shields.io/docsrs/memvid-core?style=flat-square&logo=docs.rs" alt="docs.rs" /></a>
<a href="https://github.com/memvid/memvid/blob/main/LICENSE"><img src="https://img.shields.io/badge/license-Apache%202.0-blue?style=flat-square" alt="License" /></a>
</p>
<p align="center">
<a href="https://github.com/memvid/memvid/stargazers"><img src="https://img.shields.io/github/stars/memvid/memvid?style=flat-square&logo=github" alt="Stars" /></a>
<a href="https://github.com/memvid/memvid/network/members"><img src="https://img.shields.io/github/forks/memvid/memvid?style=flat-square&logo=github" alt="Forks" /></a>
<a href="https://github.com/memvid/memvid/issues"><img src="https://img.shields.io/github/issues/memvid/memvid?style=flat-square&logo=github" alt="Issues" /></a>
<a href="https://discord.gg/2mynS7fcK7"><img src="https://img.shields.io/discord/1442910055233224745?style=flat-square&logo=discord&label=discord" alt="Discord" /></a>
</p>
<p align="center">
<a href="https://trendshift.io/repositories/17293" target="_blank"><img src="https://trendshift.io/api/badge/repositories/17293" alt="memvid%2Fmemvid | Trendshift" style="width: 250px; height: 55px;" width="250" height="55"/></a>
</p>
<!-- BADGES:END -->
<p align="center">
<strong>मेमविड AI एजेंट के लिए एक सिंगल-फाइल मेमोरी लेयर है जिसमें तुरंत रिट्रीवल और लॉन्ग-टर्म मेमोरी होती है।</strong><br/>
बिना डेटाबेस के, स्थायी, वर्शन वाली और पोर्टेबल मेमोरी।
</p>
<h2 align="center">⭐️ प्रोजेक्ट को सपोर्ट करने के लिए एक स्टार दें। ⭐️</h2>
</p>
## What is Memvid?
मेमविड एक पोर्टेबल AI मेमोरी सिस्टम है जो आपके डेटा, एम्बेडिंग, सर्च स्ट्रक्चर और मेटाडेटा को एक ही फ़ाइल में पैक करता है।
जटिल RAG पाइपलाइन या सर्वर-आधारित वेक्टर डेटाबेस चलाने के बजाय, Memvid सीधे फ़ाइल से तेज़ी से डेटा रिट्रीव करने में मदद करता है।
इसका नतीजा एक मॉडल-एग्नोस्टिक, इंफ्रास्ट्रक्चर-फ्री मेमोरी लेयर है जो AI एजेंट को परमानेंट, लॉन्ग-टर्म मेमोरी देती है जिसे वे कहीं भी ले जा सकते हैं।
---
## वीडियो फ़्रेम क्यों?
मेमविड वीडियो एन्कोडिंग से प्रेरणा लेता है, वीडियो स्टोर करने के लिए नहीं, बल्कि**AI मेमोरी को स्मार्ट फ्रेम्स के अपेंड-ओनली, अल्ट्रा-एफ़िशिएंट सीक्वेंस के तौर पर ऑर्गनाइज़ करें।**
स्मार्ट फ्रेम एक ऐसा यूनिट है जिसे बदला नहीं जा सकता, जो टाइमस्टैम्प, चेकसम और बेसिक मेटाडेटा के साथ कंटेंट स्टोर करता है।
फ्रेम को इस तरह से ग्रुप किया जाता है जिससे कुशल कम्प्रेशन, इंडेक्सिंग और पैरेलल रीड संभव हो सके।
यह फ्रेम-आधारित डिज़ाइन इन चीज़ों को संभव बनाता है:
- मौजूदा डेटा को संशोधित या दूषित किए बिना केवल डेटा जोड़ना
- पिछली मेमोरी स्थितियों पर प्रश्न
- ज्ञान कैसे विकसित होता है, इसकी टाइमलाइन-शैली में जांच
- प्रतिबद्ध, अपरिवर्तनीय फ्रेम के माध्यम से क्रैश सुरक्षा
- वीडियो एन्कोडिंग से अपनाई गई तकनीकों का उपयोग करके कुशल कम्प्रेशन।
इसका नतीजा एक सिंगल फ़ाइल होती है जो AI सिस्टम के लिए रिवाइंड करने लायक मेमोरी टाइमलाइन की तरह काम करती है।
---
## मुख्य अवधारणाएँ
- **लिविंग मेमोरी इंजन**
सेशन के दौरान लगातार मेमोरी को जोड़ें, ब्रांच करें और विकसित करें।
- **कैप्सूल संदर्भ (`.mv2`)**
नियमों और एक्सपायरी के साथ सेल्फ-कंटेन्ड, शेयर करने लायक मेमोरी कैप्सूल।
- **टाइम-ट्रैवल डिबगिंग**
किसी भी मेमोरी स्टेट को रिवाइंड, रिप्ले या ब्रांच करें।
- **स्मार्ट रिकॉल**
प्रेडिक्टिव कैशिंग के साथ सब-5ms लोकल मेमोरी एक्सेस।
- **कोडेक इंटेलिजेंस**
यह समय के साथ कम्प्रेशन को ऑटो-सेलेक्ट और अपग्रेड करता है।
---
## उपयोग के मामले
मेमविड एक पोर्टेबल, सर्वरलेस मेमोरी लेयर है जो AI एजेंट को परमानेंट मेमोरी और तेज़ रिकॉल देता है। क्योंकि यह मॉडल-एग्नोस्टिक, मल्टी-मॉडल है, और पूरी तरह से ऑफ़लाइन काम करता है, इसलिए डेवलपर्स मेमविड का इस्तेमाल कई तरह के रियल-वर्ल्ड एप्लीकेशन में कर रहे हैं।
- लंबे समय तक चलने वाले AI एजेंट
- एंटरप्राइज़ नॉलेज बेस
- ऑफ़लाइन-फ़र्स्ट AI सिस्टम
- कोडबेस को समझना
- कस्टमर सपोर्ट एजेंट
- वर्कफ़्लो ऑटोमेशन
- सेल्स और मार्केटिंग कोपायलट
- पर्सनल नॉलेज असिस्टेंट
- मेडिकल, लीगल और फाइनेंशियल एजेंट
- ऑडिट करने योग्य और डीबग करने योग्य AI वर्कफ़्लो
- कस्टम एप्लीकेशन
---
## SDKs & CLI
मेमविड को अपनी पसंदीदा भाषा में इस्तेमाल करें:
| Package | Install | Links |
| --------------- | --------------------------- | ------------------------------------------------------------------------------------------------------------------- |
| **CLI** | `npm install -g memvid-cli` | [![npm](https://img.shields.io/npm/v/memvid-cli?style=flat-square)](https://www.npmjs.com/package/memvid-cli) |
| **Node.js SDK** | `npm install @memvid/sdk` | [![npm](https://img.shields.io/npm/v/@memvid/sdk?style=flat-square)](https://www.npmjs.com/package/@memvid/sdk) |
| **Python SDK** | `pip install memvid-sdk` | [![PyPI](https://img.shields.io/pypi/v/memvid-sdk?style=flat-square)](https://pypi.org/project/memvid-sdk/) |
| **Rust** | `cargo add memvid-core` | [![Crates.io](https://img.shields.io/crates/v/memvid-core?style=flat-square)](https://crates.io/crates/memvid-core) |
---
## Installation (Rust)
### आवश्यकताएं
- **Rust 1.85.0+** — Install from [rustup.rs](https://rustup.rs)
### अपने प्रोजेक्ट में जोड़ें
```toml
[dependencies]
memvid-core = "2.0"
```
### Feature Flags
| Feature | Description |
| ------------------- | ---------------------------------------------- |
| `lex` | Full-text search with BM25 ranking (Tantivy) |
| `pdf_extract` | Pure Rust PDF text extraction |
| `vec` | Vector similarity search (HNSW + ONNX) |
| `clip` | CLIP visual embeddings for image search |
| `whisper` | Audio transcription with Whisper |
| `temporal_track` | Natural language date parsing ("last Tuesday") |
| `parallel_segments` | Multi-threaded ingestion |
| `encryption` | Password-based encryption capsules (.mv2e) |
Enable features as needed:
```toml
[dependencies]
memvid-core = { version = "2.0", features = ["lex", "vec", "temporal_track"] }
```
---
## त्वरित प्रारंभ
```rust
use memvid_core::{Memvid, PutOptions, SearchRequest};
fn main() -> memvid_core::Result<()> {
// Create a new memory file
let mut mem = Memvid::create("knowledge.mv2")?;
// Add documents with metadata
let opts = PutOptions::builder()
.title("Meeting Notes")
.uri("mv2://meetings/2024-01-15")
.tag("project", "alpha")
.build();
mem.put_bytes_with_options(b"Q4 planning discussion...", opts)?;
mem.commit()?;
// Search
let response = mem.search(SearchRequest {
query: "planning".into(),
top_k: 10,
snippet_chars: 200,
..Default::default()
})?;
for hit in response.hits {
println!("{}: {}", hit.title.unwrap_or_default(), hit.text);
}
Ok(())
}
```
---
## निर्माण
रिपॉजिटरी को क्लोन करें:
```bash
git clone https://github.com/memvid/memvid.git
cd memvid
```
डीबग मोड में बिल्ड करें:
```bash
cargo build
```
रिलीज़ मोड में बिल्ड करें (ऑप्टिमाइज़्ड):
```bash
cargo build --release
```
विशिष्ट विशेषताओं के साथ बनाएं:
```bash
cargo build --release --features "lex,vec,temporal_track"
```
---
## टेस्ट चलाएँ
सभी टेस्ट चलाएँ:
```bash
cargo test
```
आउटपुट के साथ टेस्ट चलाएँ:
```bash
cargo test -- --nocapture
```
एक विशिष्ट टेस्ट चलाएँ:
```bash
cargo test test_name
```
केवल इंटीग्रेशन टेस्ट चलाएँ:
```bash
cargo test --test lifecycle
cargo test --test search
cargo test --test mutation
```
---
## उदाहरण
The `examples/` डायरेक्टरी में काम करने वाले उदाहरण हैं:
### बेसिक उपयोग
यह क्रिएट, पुट, सर्च और टाइमलाइन ऑपरेशन दिखाता है:
```bash
cargo run --example basic_usage
```
### PDF इन्जेक्शन
PDF डॉक्यूमेंट्स को इन्जेस्ट करें और सर्च करें ("अटेंशन इज़ ऑल यू नीड" पेपर का इस्तेमाल करता है):
```bash
cargo run --example pdf_ingestion
```
### CLIP विज़ुअल सर्च
CLIP एम्बेडिंग का इस्तेमाल करके इमेज सर्च (इसके लिए `clip` फ़ीचर ज़रूरी है):
```bash
cargo run --example clip_visual_search --features clip
```
### व्हिस्पर ट्रांसक्रिप्शन
ऑडियो ट्रांसक्रिप्शन (`whisper` फीचर ज़रूरी है):
```bash
cargo run --example test_whisper --features whisper
```
---
## फ़ाइल फ़ॉर्मेट
सब कुछ एक ही `.mv2` फ़ाइल में होता है:
```
┌────────────────────────────┐
│ Header (4KB) │ Magic, version, capacity
├────────────────────────────┤
│ Embedded WAL (1-64MB) │ Crash recovery
├────────────────────────────┤
│ Data Segments │ Compressed frames
├────────────────────────────┤
│ Lex Index │ Tantivy full-text
├────────────────────────────┤
│ Vec Index │ HNSW vectors
├────────────────────────────┤
│ Time Index │ Chronological ordering
├────────────────────────────┤
│ TOC (Footer) │ Segment offsets
└────────────────────────────┘
```
कोई `.wal`, `.lock`, `.shm`, या साइडकार फ़ाइल नहीं। कभी नहीं।
पूरे फ़ाइल फ़ॉर्मेट स्पेसिफ़िकेशन के लिए [MV2_SPEC.md](MV2_SPEC.md) देखें।
---
## सपोर्ट
क्या आपके कोई सवाल या फीडबैक हैं?
Email: contact@memvid.com
**सपोर्ट दिखाने के लिए ⭐ दें**
---
## लाइसेंस
Apache License 2.0 — ज़्यादा जानकारी के लिए [LICENSE](LICENSE) फ़ाइल देखें।
+429
View File
@@ -0,0 +1,429 @@
<!-- HEADER:START -->
<img width="2000" height="524" alt="Social Cover (9)" src="https://github.com/user-attachments/assets/cf66f045-c8be-494b-b696-b8d7e4fb709c" />
<!-- HEADER:END -->
<!-- FLAGS:START -->
<p align="center">
<a href="../../README.md">🇺🇸 English</a>
<a href="README.es.md">🇪🇸 Español</a>
<a href="README.fr.md">🇫🇷 Français</a>
<a href="README.so.md">🇸🇴 Soomaali</a>
<a href="README.ar.md">🇸🇦 العربية</a>
<a href="README.nl.md">🇧🇪/🇳🇱 Nederlands</a>
<a href="README.hi.md">🇮🇳 हिन्दी</a>
<a href="README.bn.md">🇧🇩 বাংলা</a>
<a href="README.cs.md">🇨🇿 Čeština</a>
<a href="README.ko.md">🇰🇷 한국어</a>
<a href="README.ja.md">🇯🇵 日本語</a>
<!-- Next Flag -->
</p>
<!-- FLAGS:END -->
<!-- NAV:START -->
<p align="center">
<a href="https://www.memvid.com">Website</a>
·
<a href="https://sandbox.memvid.com">Try Sandbox</a>
·
<a href="https://docs.memvid.com">Docs</a>
·
<a href="https://github.com/memvid/memvid/discussions">Discussions</a>
</p>
<!-- NAV:END -->
<!-- BADGES:START -->
<p align="center">
<a href="https://crates.io/crates/memvid-core"><img src="https://img.shields.io/crates/v/memvid-core?style=flat-square&logo=rust" alt="Crates.io" /></a>
<a href="https://docs.rs/memvid-core"><img src="https://img.shields.io/docsrs/memvid-core?style=flat-square&logo=docs.rs" alt="docs.rs" /></a>
<a href="https://github.com/memvid/memvid/blob/main/LICENSE"><img src="https://img.shields.io/badge/license-Apache%202.0-blue?style=flat-square" alt="License" /></a>
</p>
<p align="center">
<a href="https://github.com/memvid/memvid/stargazers"><img src="https://img.shields.io/github/stars/memvid/memvid?style=flat-square&logo=github" alt="Stars" /></a>
<a href="https://github.com/memvid/memvid/network/members"><img src="https://img.shields.io/github/forks/memvid/memvid?style=flat-square&logo=github" alt="Forks" /></a>
<a href="https://github.com/memvid/memvid/issues"><img src="https://img.shields.io/github/issues/memvid/memvid?style=flat-square&logo=github" alt="Issues" /></a>
<a href="https://discord.gg/2mynS7fcK7"><img src="https://img.shields.io/discord/1442910055233224745?style=flat-square&logo=discord&label=discord" alt="Discord" /></a>
</p>
<p align="center">
<a href="https://trendshift.io/repositories/17293" target="_blank"><img src="https://trendshift.io/api/badge/repositories/17293" alt="memvid%2Fmemvid | Trendshift" style="width: 250px; height: 55px;" width="250" height="55"/></a>
</p>
<!-- BADGES:END -->
<p align="center">
<strong>Memvidは、AIエージェントのための即時検索と長期記憶を備えた単一ファイル型メモリレイヤーです。</strong>
</br>
データベースを必要とせず、永続化、バージョン管理、ポータブル性を備えたメモリを実現します。
</p>
<h2 align="center">⭐️ スターで応援お願いします ⭐️</h2>
</p>
## Memvidとは?
Memvidは、データ、埋め込み、検索構造、メタデータを1つのファイルにパッケージ化するポータブルAIメモリシステムです。
複雑なRAGパイプラインやサーバーベースのベクトルデータベースを運用する代わりに、Memvidを使用することで直接ファイルから高速な検索が可能になります。
その結果、モデルに依存せずインフラ不要のメモリレイヤーが実現し、AIエージェントはどこでも使える永続的な長期記憶を持つことができます。
---
## スマートフレーム (Smart Frames) とは?
Memvidは、(ビデオを保存するためではなく)**追記に特化した効率的なスマートフレームのシーケンスとしてAIメモリを整理するため**に、ビデオエンコーディング技術から着想を得ています。
スマートフレームは、コンテンツをタイムスタンプ、チェックサム、基本メタデータとともに保存する不変(イミュータブル)な単位です。フレームは効率的な圧縮、インデックス作成、並列読み取りができるようグループ化されています。
このフレームベースの設計により、以下が可能になります。
- 既存のデータを変更したり破損したりすることなくデータを追加
- 過去のメモリ状態に対するクエリ
- 知識がどのように進化するかをタイムライン形式で検査
- コミットされた不変フレームによるクラッシュ耐性
- ビデオエンコーディング技術を応用した効率的な圧縮
その結果、AIシステムの「巻き戻し可能なメモリタイムライン」のように機能する単一のファイルが生成されます。
---
## コアコンセプト
- **成長するメモリエンジン (Living Memory Engine)**
セッションをまたいでメモリを継続的に追加、分岐、進化させます。
- **カプセル・コンテキスト (`.mv2`)**
ルールや有効期限を設定できる、自己完結型で共有可能なメモリカプセル。
- **タイムトラベル・デバッグ**
任意のメモリ状態を巻き戻し、再現、または分岐させることができます。
- **スマート・リコール**
予測キャッシングによる5ミリ秒未満のローカルメモリーアクセス。
- **コーデック・インテリジェンス**
圧縮方式を自動選択し、時間の経過とともにアップグレードします。
---
## ユースケース
Memvidは、AIエージェントに永続的な記憶と高速な呼び出し機能を提供するポータブルでサーバーレスなメモリレイヤーです。モデルに依存せず、マルチモーダルに対応し、完全にオフラインで動作するため、実用的なアプリケーションで幅広く利用されています。
- 長期稼働AIエージェント
- エンタープライズ向けナレッジベース
- オフラインファーストAIシステム
- コードベースの理解
- カスタマーサポートエージェント
- ワークフロー自動化
- セールス・マーケティング支援
- パーソナル・ナレッジ・アシスタント
- 医療・法律・金融特化型エージェント
- 監査・デバッグ可能なAIワークフロー
- カスタムアプリケーション
---
## SDK と CLI
お好みの言語でMemvidを利用できます。
| パッケージ | インストール | リンク |
| --------------- | --------------------------- | ------------------------------------------------------------------------------------------------------------------- |
| **CLI** | `npm install -g memvid-cli` | [![npm](https://img.shields.io/npm/v/memvid-cli?style=flat-square)](https://www.npmjs.com/package/memvid-cli) |
| **Node.js SDK** | `npm install @memvid/sdk` | [![npm](https://img.shields.io/npm/v/@memvid/sdk?style=flat-square)](https://www.npmjs.com/package/@memvid/sdk) |
| **Python SDK** | `pip install memvid-sdk` | [![PyPI](https://img.shields.io/pypi/v/memvid-sdk?style=flat-square)](https://pypi.org/project/memvid-sdk/) |
| **Rust** | `cargo add memvid-core` | [![Crates.io](https://img.shields.io/crates/v/memvid-core?style=flat-square)](https://crates.io/crates/memvid-core) |
---
## インストール (Rust)
### 要件
- **Rust 1.85.0+** - [rustup.rs](https://rustup.rs) からインストールしてください。
### プロジェクトへの追加
```toml
[dependencies]
memvid-core = "2.0"
```
### 機能フラグ (Feature Flags)
| 機能 | 説明 |
| ------------------- | -------------------------------------------------------------- |
| `lex` | BM25ランキングによる全文検索 (Tantivy) |
| `pdf_extract` | Pure RustによるPDFテキスト抽出 |
| `vec` | ベクトル類似性検索 (HNSW + ONNXによるローカルテキスト埋め込み) |
| `clip` | 画像検索用のCLIPビジュアル埋め込み |
| `whisper` | Whisperによる音声文字起こし |
| `temporal_track` | 自然言語による日付解析 (例: "last Tuesday") |
| `parallel_segments` | マルチスレッドによるデータ取り込み |
| `encryption` | パスワードベースの暗号化カプセル (.mv2e) |
以下のように、必要に応じて有効化してください。
```toml
[dependencies]
memvid-core = { version = "2.0", features = ["lex", "vec", "temporal_track"] }
```
---
## クイックスタート
```rust
use memvid_core::{Memvid, PutOptions, SearchRequest};
fn main() -> memvid_core::Result<()> {
// 新しいメモリファイルを作成
let mut mem = Memvid::create("knowledge.mv2")?;
// メタデータ付きでドキュメントを追加
let opts = PutOptions::builder()
.title("Meeting Notes")
.uri("mv2://meetings/2024-01-15")
.tag("project", "alpha")
.build();
mem.put_bytes_with_options(b"Q4 planning discussion...", opts)?;
mem.commit()?;
// 検索の実行
let response = mem.search(SearchRequest {
query: "planning".into(),
top_k: 10,
snippet_chars: 200,
..Default::default()
})?;
for hit in response.hits {
println!("{}: {}", hit.title.unwrap_or_default(), hit.text);
}
Ok(())
}
```
---
## ビルド
リポジトリをクローン:
```bash
git clone https://github.com/memvid/memvid.git
cd memvid
```
デバッグモードでビルド:
```bash
cargo build
```
リリースモードでビルド(最適化):
```bash
cargo build --release
```
特定の機能フラグ付きでビルド:
```bash
cargo build --release --features "lex,vec,temporal_track"
```
---
## テストの実行
すべてのテストを実行:
```bash
cargo test
```
標準出力でテストを実行:
```bash
cargo test -- --nocapture
```
特定のテストを実行:
```bash
cargo test test_name
```
統合テストのみを実行:
```bash
cargo test --test lifecycle
cargo test --test search
cargo test --test mutation
```
---
## サンプル (Examples)
`examples/` ディレクトリには、実際に動作するサンプルコードが用意されています。
### 基本的な使い方 (Basic Usage)
作成 (create)、追加 (put)、検索 (search)、およびタイムライン操作のデモです。
```bash
cargo run --example basic_usage
```
### PDFの取り込み (PDF Ingestion)
PDFドキュメントの取り込みと検索のサンプルです。(論文「Attention Is All You Need」を使用)
```bash
cargo run --example pdf_ingestion
```
### CLIPによる画像検索 (CLIP Visual Search)
CLIP埋め込みを使用した画像検索のサンプルです。
```bash
cargo run --example clip_visual_search --features clip
```
### Whisperによる文字起こし (Whisper Transcription)
音声文字起こしのサンプルです。
```bash
cargo run --example test_whisper --features whisper
```
---
## テキスト埋め込みモデル
`vec` 機能は、ONNXモデルを使用したローカルでのテキスト埋め込みをサポートしています。利用前にモデルファイルを手動でダウンロードする必要があります。
### 推奨:BGE-small (デフォルト)
高速で効率的なBGE-smallモデル(384次元)をダウンロードします。
```bash
mkdir -p ~/.cache/memvid/text-models
# ONNXモデルのダウンロード
curl -L 'https://huggingface.co/BAAI/bge-small-en-v1.5/resolve/main/onnx/model.onnx' \
-o ~/.cache/memvid/text-models/bge-small-en-v1.5.onnx
# トークナイザーのダウンロード
curl -L 'https://huggingface.co/BAAI/bge-small-en-v1.5/resolve/main/tokenizer.json' \
-o ~/.cache/memvid/text-models/bge-small-en-v1.5_tokenizer.json
```
### モデル一覧
| モデル | 次元数 | サイズ | 最適な用途 |
| ----------------------- | ------ | ------ | ------------------------ |
| `bge-small-en-v1.5` | 384 | ~120MB | デフォルト(高速・軽量) |
| `bge-base-en-v1.5` | 768 | ~420MB | より高い精度が必要な場合 |
| `nomic-embed-text-v1.5` | 768 | ~530MB | 多目的なタスク |
| `gte-large` | 1024 | ~1.3GB | 最高精度 |
### 他のモデル
**BGE-base** (768次元):
```bash
curl -L 'https://huggingface.co/BAAI/bge-base-en-v1.5/resolve/main/onnx/model.onnx' \
-o ~/.cache/memvid/text-models/bge-base-en-v1.5.onnx
curl -L 'https://huggingface.co/BAAI/bge-base-en-v1.5/resolve/main/tokenizer.json' \
-o ~/.cache/memvid/text-models/bge-base-en-v1.5_tokenizer.json
```
**Nomic** (768次元):
```bash
curl -L 'https://huggingface.co/nomic-ai/nomic-embed-text-v1.5/resolve/main/onnx/model.onnx' \
-o ~/.cache/memvid/text-models/nomic-embed-text-v1.5.onnx
curl -L 'https://huggingface.co/nomic-ai/nomic-embed-text-v1.5/resolve/main/tokenizer.json' \
-o ~/.cache/memvid/text-models/nomic-embed-text-v1.5_tokenizer.json
```
**GTE-large** (1024次元):
```bash
curl -L 'https://huggingface.co/thenlper/gte-large/resolve/main/onnx/model.onnx' \
-o ~/.cache/memvid/text-models/gte-large.onnx
curl -L 'https://huggingface.co/thenlper/gte-large/resolve/main/tokenizer.json' \
-o ~/.cache/memvid/text-models/gte-large_tokenizer.json
```
### 使用例
```rust
use memvid_core::text_embed::{LocalTextEmbedder, TextEmbedConfig};
use memvid_core::types::embedding::EmbeddingProvider;
// デフォルトモデルを使用する場合 (BGE-small)
let config = TextEmbedConfig::default();
let embedder = LocalTextEmbedder::new(config)?;
let embedding = embedder.embed_text("hello world")?;
assert_eq!(embedding.len(), 384);
// モデルを変更する場合
let config = TextEmbedConfig::bge_base();
let embedder = LocalTextEmbedder::new(config)?;
```
類似性の計算と検索ランキングを含む完全な例については、`examples/text_embedding.rs` を参照してください。
---
## ファイル構成
すべてが単一の `.mv2` ファイルに収められます。
```
┌────────────────────────────┐
│ ヘッダー (4KB) │ マジックナンバー、バージョン、容量
├────────────────────────────┤
│ 組み込みWAL (1-64MB) │ クラッシュリカバリ用
├────────────────────────────┤
│ データセグメント │ 圧縮されたフレーム
├────────────────────────────┤
│ 全文検索インデックス (Lex) │ Tantivy全文検索
├────────────────────────────┤
│ ベクトルインデックス (Vec) │ HNSWベクトル
├────────────────────────────┤
│ タイムインデックス │ 時系列順序
├────────────────────────────┤
│ TOC (フッター) │ セグメントオフセット
└────────────────────────────┘
```
`.wal``.lock``.shm` などのサイドカーファイルは一切生成されません。
フォーマット仕様の詳細は [MV2_SPEC.md](MV2_SPEC.md) を参照してください。
---
## サポート
ご質問やフィードバックはこちらまでご連絡ください。
メール: contact@memvid.com
**⭐でプロジェクトをサポートしてください。**
---
## ライセンス
Apache License 2.0 - 詳細は [LICENSE](LICENSE) ファイルをご覧ください。
+424
View File
@@ -0,0 +1,424 @@
<!-- HEADER:START -->
<img width="2000" height="524" alt="Social Cover (9)" src="https://github.com/user-attachments/assets/cf66f045-c8be-494b-b696-b8d7e4fb709c" />
<!-- HEADER:END -->
<!-- FLAGS:START -->
<p align="center">
<a href="../../README.md">🇺🇸 English</a>
<a href="README.es.md">🇪🇸 Español</a>
<a href="README.fr.md">🇫🇷 Français</a>
<a href="README.so.md">🇸🇴 Soomaali</a>
<a href="README.ar.md">🇸🇦 العربية</a>
<a href="README.nl.md">🇧🇪/🇳🇱 Nederlands</a>
<a href="README.hi.md">🇮🇳 हिन्दी</a>
<a href="README.bn.md">🇧🇩 বাংলা</a>
<a href="README.cs.md">🇨🇿 Čeština</a>
<a href="README.ko.md">🇰🇷 한국어</a>
<a href="README.ja.md">🇯🇵 日本語</a>
<!-- Next Flag -->
</p>
<!-- FLAGS:END -->
<!-- NAV:START -->
<p align="center">
<a href="https://www.memvid.com">Website</a>
·
<a href="https://sandbox.memvid.com">Try Sandbox</a>
·
<a href="https://docs.memvid.com">Docs</a>
·
<a href="https://github.com/memvid/memvid/discussions">Discussions</a>
</p>
<!-- NAV:END -->
<!-- BADGES:START -->
<p align="center">
<a href="https://crates.io/crates/memvid-core"><img src="https://img.shields.io/crates/v/memvid-core?style=flat-square&logo=rust" alt="Crates.io" /></a>
<a href="https://docs.rs/memvid-core"><img src="https://img.shields.io/docsrs/memvid-core?style=flat-square&logo=docs.rs" alt="docs.rs" /></a>
<a href="https://github.com/memvid/memvid/blob/main/LICENSE"><img src="https://img.shields.io/badge/license-Apache%202.0-blue?style=flat-square" alt="License" /></a>
</p>
<p align="center">
<a href="https://github.com/memvid/memvid/stargazers"><img src="https://img.shields.io/github/stars/memvid/memvid?style=flat-square&logo=github" alt="Stars" /></a>
<a href="https://github.com/memvid/memvid/network/members"><img src="https://img.shields.io/github/forks/memvid/memvid?style=flat-square&logo=github" alt="Forks" /></a>
<a href="https://github.com/memvid/memvid/issues"><img src="https://img.shields.io/github/issues/memvid/memvid?style=flat-square&logo=github" alt="Issues" /></a>
<a href="https://discord.gg/2mynS7fcK7"><img src="https://img.shields.io/discord/1442910055233224745?style=flat-square&logo=discord&label=discord" alt="Discord" /></a>
</p>
<p align="center">
<a href="https://trendshift.io/repositories/17293" target="_blank"><img src="https://trendshift.io/api/badge/repositories/17293" alt="memvid%2Fmemvid | Trendshift" style="width: 250px; height: 55px;" width="250" height="55"/></a>
</p>
<!-- BADGES:END -->
<p align="center">
<strong>Memvid는 AI 에이전트를 위한 단일 파일 메모리 레이어로, 인스턴스 검색 및 장기 메모리 기능을 제공합니다.</strong><br/>
데이터 베이스 없이 지속적이고, 버전 관리가 용이하며 여러 어플리케이션에 자유로운 적용이 가능합니다.
</p>
<h2 align="center">⭐️ STAR로 이 프로젝트를 지원해주세요 ⭐️</h2>
</p>
## Memvid란?
Memvid는 데이터, 임베딩, 검색 구조 및 메타데이터를 단일 파일로 패키징하는 이식 가능한 AI 메모리 시스템입니다.
복잡한 RAG 파이프라인이나 서버 기반 벡터 데이터베이스를 실행하는 대신, Memvid는 파일에서 직접 빠른 검색을 가능하게 합니다.
결과적으로 모델에 독립적이며 인프라 구조와는 독립적인 메모리 레이어로, AI 에이전트가 어디서나 휴대할 수 있는 지속적 장기 메모리를 제공합니다
---
## Smart Frames란?
Memvid는 **AI 메모리를 추가 전용(append-only)의 초고효율 Smart Frame 시퀀스로 구성하기 위해** 비디오 인코딩에서 영감을 받았습니다.
Smart Frame은 타임스탬프, 체크섬 및 기본 메타데이터와 함께 콘텐츠를 저장하는 불변 단위입니다.
프레임은 효율적인 압축, 인덱싱 및 병렬 읽기를 허용하는 방식으로 그룹화됩니다.
이러한 프레임 기반 설계는 다음을 가능하게 합니다:
- 기존 데이터를 수정하거나 손상시키지 않는 추가 전용(append-only) 쓰기
- 과거 메모리 상태에 대한 쿼리
- 지식이 어떻게 변화하는지에 대한 타임라인 스타일 검사
- 불변 프레임워크를 통한 크래시 안전성
- 비디오 인코딩에서 차용한 기술을 사용한 효율적인 압축
이를 위한 결과물은 AI 시스템을 위한 되감기 가능한 메모리 타임라인처럼 동작하는 단일 파일입니다.
---
## 주요 개념
- **실시간 변화하는 메모리 엔진**
세션 간에 메모리를 지속적으로 추가, 분기 및 변화시킵니다.
- **문맥 캡슐화 (`.mv2`)**
규칙과 만료 시간이 포함된 자립형 공유 가능 형대의 메모리 캡슐입니다.
- **시간 기반 디버깅**
임의의 메모리 상태로 되감기, 재생 또는 분기합니다.
- **예측 기반 호출**
예측 캐싱을 사용한 5ms 미만 로컬 메모리 액세스를 제공합니다.
- **코덱 인텔리전스**
시간 경과에 따라 압축을 자동 선택 및 업그레이드합니다.
---
## 이용 사례
Memvid 이동 가능한 서버리스 메모리 레이어로 AI 에이전트에 지속적인 메모리와 빠른 호출을 제공합니다. 이는 모델과 독립적이고, 멀티모달을 지원하며, 인터넷을 사용하지 않으므로, 개발자들은 다양한 실제 어플리케이션에서 Memvid를 활용하고 있습니다.
- 장기 실행 AI 에이전트
- 기업 내의 지식 베이스
- 오프라인 우선의 AI 시스템
- 코드베이스 이해
- 고객 지원 에이전트
- 워크플로 자동화
- 판매 및 마케팅 코파일럿
- 개인 지식 어시스턴트
- 의료, 법률 및 금융 에이전트
- 모니터링 및 디버깅 가능한 AI 워크플로
- 그 외의 여러 애플리케이션
---
## SDKs & CLI
원하는 언어로 Memvid를 사용하세요:
| 패키지 | 설치 커맨드 | 링크 |
| --------------- | --------------------------- | ------------------------------------------------------------------------------------------------------------------- |
| **CLI** | `npm install -g memvid-cli` | [![npm](https://img.shields.io/npm/v/memvid-cli?style=flat-square)](https://www.npmjs.com/package/memvid-cli) |
| **Node.js SDK** | `npm install @memvid/sdk` | [![npm](https://img.shields.io/npm/v/@memvid/sdk?style=flat-square)](https://www.npmjs.com/package/@memvid/sdk) |
| **Python SDK** | `pip install memvid-sdk` | [![PyPI](https://img.shields.io/pypi/v/memvid-sdk?style=flat-square)](https://pypi.org/project/memvid-sdk/) |
| **Rust** | `cargo add memvid-core` | [![Crates.io](https://img.shields.io/crates/v/memvid-core?style=flat-square)](https://crates.io/crates/memvid-core) |
---
## 설치 (Rust)
### 요구 사항
- **Rust 1.85.0+** — [rustup.rs](https://rustup.rs)에서 설치 가능합니다.
### 프로젝트에 추가
```toml
[dependencies]
memvid-core = "2.0"
```
### Feature Flags
| Feature | Description |
| ------------------- | --------------------------------------------------- |
| `lex` | BM25 랭킹 기반 전체 텍스트 검색 (Tantivy) |
| `pdf_extract` | Rust 기반 PDF 텍스트 추출 |
| `vec` | 벡터 유사도 검색 (HNSW + ONNX) |
| `clip` | 이미지 검색을 위한 CLIP 임베딩 |
| `whisper` | Whisper 기반 오디오 전사 |
| `temporal_track` | 자연어 날짜 추출 ("지난 화요일") |
| `parallel_segments` | 멀티-스레딩 처리 |
| `encryption` | Password 기반 암호화 (.mv2e) |
필요한 기능을 아래 방식으로 활성화하세요:
```toml
[dependencies]
memvid-core = { version = "2.0", features = ["lex", "vec", "temporal_track"] }
```
---
## Quick Start
```rust
use memvid_core::{Memvid, PutOptions, SearchRequest};
fn main() -> memvid_core::Result<()> {
// Create a new memory file
let mut mem = Memvid::create("knowledge.mv2")?;
// Add documents with metadata
let opts = PutOptions::builder()
.title("Meeting Notes")
.uri("mv2://meetings/2024-01-15")
.tag("project", "alpha")
.build();
mem.put_bytes_with_options(b"Q4 planning discussion...", opts)?;
mem.commit()?;
// Search
let response = mem.search(SearchRequest {
query: "planning".into(),
top_k: 10,
snippet_chars: 200,
..Default::default()
})?;
for hit in response.hits {
println!("{}: {}", hit.title.unwrap_or_default(), hit.text);
}
Ok(())
}
```
---
## 빌드
이 레포지토리 클론:
```bash
git clone https://github.com/memvid/memvid.git
cd memvid
```
디버그 모드로 빌드:
```bash
cargo build
```
배포 모드로 빌드 (optimized):
```bash
cargo build --release
```
특수 기능을 포함하도록 빌드:
```bash
cargo build --release --features "lex,vec,temporal_track"
```
---
## 테스트
전체 테스트 실행:
```bash
cargo test
```
테스트 실행 및 결과 출력:
```bash
cargo test -- --nocapture
```
특정 테스트 실행:
```bash
cargo test test_name
```
인테그레이션 테스트만 실행:
```bash
cargo test --test lifecycle
cargo test --test search
cargo test --test mutation
```
---
## 예시
`examples/` 디렉토리에 예제가 있습니다:
### 기본 사용법
생성, 추가, 검색 및 타임라인 작업을 보여줍니다:
```bash
cargo run --example basic_usage
```
### PDF 수집
PDF 문서 수집 및 검색 ("Attention Is All You Need" 논문 사용):
```bash
cargo run --example pdf_ingestion
```
### CLIP 이미지 검색
CLIP 임베딩을 사용한 이미지 검색 (`clip` 기능 필요):
```bash
cargo run --example clip_visual_search --features clip
```
### Whisper 전사
오디오 전사 (`whisper` 기능 필요):
```bash
cargo run --example test_whisper --features whisper
```
---
## Text Embedding 모델
`vec` 기능은 ONNX 모델을 사용한 로컬 텍스트 임베딩을 포함합니다. 로컬 텍스트 임베딩을 사용하기 전에 모델 파일을 수동으로 다운로드해야 합니다.
### Quick Start: BGE-small (추천함)
기본 BGE-small 모델(384 차원, 빠르고 효율적) 다운로드:
```bash
mkdir -p ~/.cache/memvid/text-models
# Download ONNX model
curl -L 'https://huggingface.co/BAAI/bge-small-en-v1.5/resolve/main/onnx/model.onnx' \
-o ~/.cache/memvid/text-models/bge-small-en-v1.5.onnx
# Download tokenizer
curl -L 'https://huggingface.co/BAAI/bge-small-en-v1.5/resolve/main/tokenizer.json' \
-o ~/.cache/memvid/text-models/bge-small-en-v1.5_tokenizer.json
```
### 지원 모델
| 모델명 | 차원 수 | 크기 | 권장 용도 |
| ----------------------- | ---------- | ----- | --------------------- |
| `bge-small-en-v1.5` | 384 | ~120MB | 기본 설정, 가장 빠름 |
| `bge-base-en-v1.5` | 768 | ~420MB | 꽤 좋은 성능 |
| `nomic-embed-text-v1.5` | 768 | ~530MB | 다양한 업무 가능 |
| `gte-large` | 1024 | ~1.3GB | 가장 좋은 성능 |
### 타 모델
**BGE-base** (768 dimensions):
```bash
curl -L 'https://huggingface.co/BAAI/bge-base-en-v1.5/resolve/main/onnx/model.onnx' \
-o ~/.cache/memvid/text-models/bge-base-en-v1.5.onnx
curl -L 'https://huggingface.co/BAAI/bge-base-en-v1.5/resolve/main/tokenizer.json' \
-o ~/.cache/memvid/text-models/bge-base-en-v1.5_tokenizer.json
```
**Nomic** (768 dimensions):
```bash
curl -L 'https://huggingface.co/nomic-ai/nomic-embed-text-v1.5/resolve/main/onnx/model.onnx' \
-o ~/.cache/memvid/text-models/nomic-embed-text-v1.5.onnx
curl -L 'https://huggingface.co/nomic-ai/nomic-embed-text-v1.5/resolve/main/tokenizer.json' \
-o ~/.cache/memvid/text-models/nomic-embed-text-v1.5_tokenizer.json
```
**GTE-large** (1024 dimensions):
```bash
curl -L 'https://huggingface.co/thenlper/gte-large/resolve/main/onnx/model.onnx' \
-o ~/.cache/memvid/text-models/gte-large.onnx
curl -L 'https://huggingface.co/thenlper/gte-large/resolve/main/tokenizer.json' \
-o ~/.cache/memvid/text-models/gte-large_tokenizer.json
```
### 코드 내 사용법
```rust
use memvid_core::text_embed::{LocalTextEmbedder, TextEmbedConfig};
use memvid_core::types::embedding::EmbeddingProvider;
// Use default model (BGE-small)
let config = TextEmbedConfig::default();
let embedder = LocalTextEmbedder::new(config)?;
let embedding = embedder.embed_text("hello world")?;
assert_eq!(embedding.len(), 384);
// Use different model
let config = TextEmbedConfig::bge_base();
let embedder = LocalTextEmbedder::new(config)?;
```
유사도 계산 및 검색 랭킹이 포함된 전체 예제는 `examples/text_embedding.rs`를 참조하세요.
---
## 파일 구조
모든 구성 요소는 단일 `.mv2` 파일 내에 구성됩니다:
```
┌────────────────────────────┐
│ Header (4KB) │ Magic, version, capacity
├────────────────────────────┤
│ Embedded WAL (1-64MB) │ Crash recovery
├────────────────────────────┤
│ Data Segments │ Compressed frames
├────────────────────────────┤
│ Lex Index │ Tantivy full-text
├────────────────────────────┤
│ Vec Index │ HNSW vectors
├────────────────────────────┤
│ Time Index │ Chronological ordering
├────────────────────────────┤
│ TOC (Footer) │ Segment offsets
└────────────────────────────┘
```
`.wal`, `.lock`, `.shm`, 혹은 그 외의 별도 구성 요소는 없습니다.
[MV2_SPEC.md](MV2_SPEC.md)에서 파일 세부 형식을 확인할 수 있습니다.
---
## Support
문의 사항은 아래 이메일로 부탁드립니다.
Email: contact@memvid.com
**⭐를 눌러 이 프로젝트를 지원해주세요**
---
## License
Apache License 2.0 — [LICENSE](LICENSE) 파일 참고.
+90
View File
@@ -0,0 +1,90 @@
# Internationalization (i18n) - README Translations
This folder contains translated versions of the main [README.md](../../README.md) file for different languages.
## Purpose
The `docs/i18n/` directory is dedicated to making Memvid accessible to developers worldwide by providing localized versions of the main README. Each translation helps non-English speakers understand and use Memvid more effectively.
## Structure
```
docs/i18n/
├── README.md # This file
├── README.zh-CN.md # Chinese (Simplified) translation
├── README.zh-TW.md # Chinese (Traditional) translation
├── README.es.md # Spanish translation
├── README.fr.md # French translation
├── README.de.md # German translation
├── README.ja.md # Japanese translation
├── README.ko.md # Korean translation
├── README.pt-BR.md # Portuguese (Brazil) translation
├── README.so.md # Somali translation
└── ... # Additional languages
```
## Contributing Translations
We welcome contributions of README translations! For detailed guidelines, see [Contributing Translations](CONTRIBUTING_TRANSLATIONS.md).
Here's a quick overview:
### 1. Check Existing Translations
Before starting, check if a translation for your language already exists or is in progress.
### 2. Create a Translation File
- Use the format: `README.{language-code}.md`
- Use standard language codes (e.g., `zh-CN`, `es`, `fr`, `de`, `ja`, `ko`, `pt-BR`, `so`)
- Copy the main `README.md` as a starting point
### 3. Translation Guidelines
- **Keep the structure**: Maintain the same headings, sections, and formatting as the original
- **Preserve links**: Keep all URLs and links unchanged
- **Preserve code blocks**: Keep code examples, commands, and technical terms in English (or add comments in the target language)
- **Update badges**: Keep badges and shields as they are (they're language-agnostic)
- **Maintain accuracy**: Ensure technical accuracy while making the content natural in the target language
### 4. Submit Your Translation
1. Create a new file: `README.{language-code}.md` in this directory
2. Translate the content while following the guidelines above
3. Submit a pull request with:
- A clear description of the language being added
- Your name/username for attribution (if desired)
## Language Codes
Use standard ISO 639-1 or ISO 639-2 language codes:
- `zh-CN` - Chinese (Simplified)
- `zh-TW` - Chinese (Traditional)
- `es` - Spanish
- `fr` - French
- `de` - German
- `ja` - Japanese
- `ko` - Korean
- `pt-BR` - Portuguese (Brazil)
- `ru` - Russian
- `ar` - Arabic
- `hi` - Hindi
- `so` - Somali
- And more...
## Maintenance
Translations should be updated when the main README is significantly changed. Contributors are encouraged to keep their translations in sync with the English version.
## Questions?
If you have questions about translations or want to coordinate with other translators, please:
- See [Contributing Translations](CONTRIBUTING_TRANSLATIONS.md) for detailed guidelines
- Open an issue on GitHub
- Join our [Discussions](https://github.com/memvid/memvid/discussions)
- Contact: contact@memvid.com
---
**Thank you for helping make Memvid accessible to developers worldwide! 🌍**
+347
View File
@@ -0,0 +1,347 @@
<!-- HEADER:START -->
<img width="2000" height="524" alt="Social Cover (9)" src="https://github.com/user-attachments/assets/cf66f045-c8be-494b-b696-b8d7e4fb709c" />
<!-- HEADER:END -->
<!-- FLAGS:START -->
<p align="center">
<a href="../../README.md">🇺🇸 English</a>
<a href="README.es.md">🇪🇸 Español</a>
<a href="README.fr.md">🇫🇷 Français</a>
<a href="README.so.md">🇸🇴 Soomaali</a>
<a href="README.ar.md">🇸🇦 العربية</a>
<a href="README.nl.md">🇧🇪/🇳🇱 Nederlands</a>
<a href="README.hi.md">🇮🇳 हिन्दी</a>
<a href="README.bn.md">🇧🇩 বাংলা</a>
<a href="README.cs.md">🇨🇿 Čeština</a>
<a href="README.ko.md">🇰🇷 한국어</a>
<a href="README.ja.md">🇯🇵 日本語</a>
<!-- Next Flag -->
</p>
<!-- FLAGS:END -->
<!-- NAV:START -->
<p align="center">
<a href="https://www.memvid.com">Website</a>
·
<a href="https://sandbox.memvid.com">Try Sandbox</a>
·
<a href="https://docs.memvid.com">Docs</a>
·
<a href="https://github.com/memvid/memvid/discussions">Discussions</a>
</p>
<!-- NAV:END -->
<!-- BADGES:START -->
<p align="center">
<a href="https://crates.io/crates/memvid-core"><img src="https://img.shields.io/crates/v/memvid-core?style=flat-square&logo=rust" alt="Crates.io" /></a>
<a href="https://docs.rs/memvid-core"><img src="https://img.shields.io/docsrs/memvid-core?style=flat-square&logo=docs.rs" alt="docs.rs" /></a>
<a href="https://github.com/memvid/memvid/blob/main/LICENSE"><img src="https://img.shields.io/badge/license-Apache%202.0-blue?style=flat-square" alt="License" /></a>
</p>
<p align="center">
<a href="https://github.com/memvid/memvid/stargazers"><img src="https://img.shields.io/github/stars/memvid/memvid?style=flat-square&logo=github" alt="Stars" /></a>
<a href="https://github.com/memvid/memvid/network/members"><img src="https://img.shields.io/github/forks/memvid/memvid?style=flat-square&logo=github" alt="Forks" /></a>
<a href="https://github.com/memvid/memvid/issues"><img src="https://img.shields.io/github/issues/memvid/memvid?style=flat-square&logo=github" alt="Issues" /></a>
<a href="https://discord.gg/2mynS7fcK7"><img src="https://img.shields.io/discord/1442910055233224745?style=flat-square&logo=discord&label=discord" alt="Discord" /></a>
</p>
<p align="center">
<a href="https://trendshift.io/repositories/17293" target="_blank"><img src="https://trendshift.io/api/badge/repositories/17293" alt="memvid%2Fmemvid | Trendshift" style="width: 250px; height: 55px;" width="250" height="55"/></a>
</p>
<!-- BADGES:END -->
<p align="center">
<strong>Memvid is een geheugenlaag van één bestand voor AI-agenten met directe toegang en langetermijnsgeheugen.</strong><br/>
Volhardend en draagbaar geheugen met versiebeheer en zonder databases.
</p>
<h2 align="center">⭐️ Laat een ster achter om het project te steunen ⭐️</h2>
</p>
## Wat is Memvid?
Memvid is een draagbaar AI-geheugensysteem dat uw data, embedding, zoekstructuur en metadata in één bestand opslaat.
In plaats van complexe RAG pijplijnen of servergebaseerde vectordatabases te gebruiken, zal Memvid snelle toegang recht vanuit het bestand toestaan.
Het resultaat is een model-agnostische, infrastructuurvrije geheugenlaag die AI-agenten een volhardende langetermijnsgeheugen geeft, die ze overal kunnen meenemen.
---
## Waarom videoframes?
Memvid neemt inspiratie uit videos encoderen, niet om de video op te slaan, maar om **het organiseren van AI-geheugen als een ultra-efficiënte sequentie van Smart Frames waarbij je enkel kan toevoegen.**
Een Smart Frame is een immutabele eenheid die content opslaat samen met zijn tijdstempels, controlesommen en basismetadata.
Frames worden gegroupeerd in een manier die voor efficiënte compressie, indexing en parallele lezingen zorgt.
Dit frame-gebaseerde design maakt het volgende mogelijk:
- Append-only bijschrijven van data zonder het aanpassen of corrumperen van bestaande data
- Zoekopdrachten over vorige geheugenstaten
- Tijdlijn-stijl inspectie van hoe kennis evolueert
- Crashveiligheid door de vastgelegde immutabele frames
- Efficiënte compressie gebruikmakend van technieken aangepast uit video encoderen
Het resultaat is één bestand dat werkt als een terugspoelbare geheugentijdslijn van AI-systemen.
---
## Basisconcepten
- **Living Memory Engine**
Append, vertakt en evolueert geheugen continu over sessies.
- **Capsule Context (`.mv2`)**
Autonome, deelbaar geheugencapsules met regels en vervalling.
- **Time-Travel Debugging**
Spoel terug, herspeel, of vertak elke geheugenstatus.
- **Smart Recall**
Sub-5ms lokale geheugentoegang met voorspelbare caching.
- **Codec Intelligence**
Selecteert en verbetert automatisch de compressie doorheen de tijd.
---
## Gebruiksgevallen
Memvid is een draagbare, serverloze geheugenlaag dat AI-agenten een volhardend geheugen en snelle herroepingen geeft. Door zijn model-agnostische, multi-modale en het feit dat het volledig offline werkt, gebruiken ontwikkelaars het over een wijd scala aan real-world applicaties.
- Lang werkende AI-agenten
- Kennisbanken voor ondernemingen
- Offline-First AI-systemen
- Codebase-begrip
- Klantenondersteuningsagenten
- Automatisering van de workflow
- Verkoop- en marketingcopiloten
- Persoonlijke Kennisassistenten
- Medische, juridische en financiële adviseurs
- Controleerbare en debugbare AI-workflows
- Aangepaste toepassingen
---
## SDKs & CLI
Gebruik Memvid in je lievelingstaal:
| Pakket | Installatie | Links |
| --------------- | --------------------------- | ------------------------------------------------------------------------------------------------------------------- |
| **CLI** | `npm install -g memvid-cli` | [![npm](https://img.shields.io/npm/v/memvid-cli?style=flat-square)](https://www.npmjs.com/package/memvid-cli) |
| **Node.js SDK** | `npm install @memvid/sdk` | [![npm](https://img.shields.io/npm/v/@memvid/sdk?style=flat-square)](https://www.npmjs.com/package/@memvid/sdk) |
| **Python SDK** | `pip install memvid-sdk` | [![PyPI](https://img.shields.io/pypi/v/memvid-sdk?style=flat-square)](https://pypi.org/project/memvid-sdk/) |
| **Rust** | `cargo add memvid-core` | [![Crates.io](https://img.shields.io/crates/v/memvid-core?style=flat-square)](https://crates.io/crates/memvid-core) |
---
## Installatie (Rust)
### Benodigdheden
- **Rust 1.85.0+** — Installeer vanuit [rustup.rs](https://rustup.rs)
### Voeg dit aan je project toe
```toml
[dependencies]
memvid-core = "2.0"
```
### Feature Flags
| Feature | Beschrijving |
| ------------------- | ---------------------------------------------- |
| `lex` | Full-text search with BM25 ranking (Tantivy) |
| `pdf_extract` | Pure Rust PDF text extraction |
| `vec` | Vector similarity search (HNSW + ONNX) |
| `clip` | CLIP visual embeddings for image search |
| `whisper` | Audio transcription with Whisper |
| `temporal_track` | Natural language date parsing ("last Tuesday") |
| `parallel_segments` | Multi-threaded ingestion |
| `encryption` | Password-based encryption capsules (.mv2e) |
Schakel functies in indien nodig:
```toml
[dependencies]
memvid-core = { version = "2.0", features = ["lex", "vec", "temporal_track"] }
```
---
## Quick Start
```rust
use memvid_core::{Memvid, PutOptions, SearchRequest};
fn main() -> memvid_core::Result<()> {
// Create a new memory file
let mut mem = Memvid::create("knowledge.mv2")?;
// Add documents with metadata
let opts = PutOptions::builder()
.title("Meeting Notes")
.uri("mv2://meetings/2024-01-15")
.tag("project", "alpha")
.build();
mem.put_bytes_with_options(b"Q4 planning discussion...", opts)?;
mem.commit()?;
// Search
let response = mem.search(SearchRequest {
query: "planning".into(),
top_k: 10,
snippet_chars: 200,
..Default::default()
})?;
for hit in response.hits {
println!("{}: {}", hit.title.unwrap_or_default(), hit.text);
}
Ok(())
}
```
---
## Build
Clone de repository:
```bash
git clone https://github.com/memvid/memvid.git
cd memvid
```
Build in debug modus:
```bash
cargo build
```
Build in release modus (geoptimaliseerd):
```bash
cargo build --release
```
Build with specific features:
```bash
cargo build --release --features "lex,vec,temporal_track"
```
---
## Tests uitvoeren
Voer alle tests uit:
```bash
cargo test
```
Voer tests uit met uitvoer:
```bash
cargo test -- --nocapture
```
Voer een specifieke test uit:
```bash
cargo test test_name
```
Voer enkel integratie tests uit:
```bash
cargo test --test lifecycle
cargo test --test search
cargo test --test mutation
```
---
## Voorbeelden
De `examples/` map bedraagd werkende Voorbeelden:
### Basisgebruik
Beeldt create, put, search, and timeline operaties uit:
```bash
cargo run --example basic_usage
```
### PDF Ingestion
PDF-documenten importeren en doorzoeken (gebruikt de "Attention Is All You Need" paper):
```bash
cargo run --example pdf_ingestion
```
### CLIP Visual Search
Afbeeldingen zoeken met behulp van CLIP-integraties (gebruikt `clip` feature):
```bash
cargo run --example clip_visual_search --features clip
```
### Whisper Transcription
Audio transcripties (gebruikt `whisper` feature):
```bash
cargo run --example test_whisper --features whisper
```
---
## Bestandsformaat
Alles leeft in één `.mv2` bestand:
```
┌────────────────────────────┐
│ Header (4KB) │ Magic, version, capacity
├────────────────────────────┤
│ Embedded WAL (1-64MB) │ Crash recovery
├────────────────────────────┤
│ Data Segments │ Compressed frames
├────────────────────────────┤
│ Lex Index │ Tantivy full-text
├────────────────────────────┤
│ Vec Index │ HNSW vectors
├────────────────────────────┤
│ Time Index │ Chronological ordering
├────────────────────────────┤
│ TOC (Footer) │ Segment offsets
└────────────────────────────┘
```
Geen `.wal`, `.lock`, `.shm`, of sidecar-bestanden. Ooit.
Zie [MV2_SPEC.md](MV2_SPEC.md) voor de complete bestandsformaat specificaties.
---
## Ondersteuning
Heb je vragen of feedback?
Email: contact@memvid.com
**Laat een ⭐ om je ondersteuning te tonen**
---
## Licentie
Apache License 2.0 — zie het [LICENSE](LICENSE) bestandvoor details.
+345
View File
@@ -0,0 +1,345 @@
<!-- HEADER:START -->
<img width="2000" height="524" alt="Social Cover (9)" src="https://github.com/user-attachments/assets/cf66f045-c8be-494b-b696-b8d7e4fb709c" />
<!-- HEADER:END -->
<!-- FLAGS:START -->
<p align="center">
<a href="../../README.md">🇺🇸 English</a>
<a href="README.es.md">🇪🇸 Español</a>
<a href="README.fr.md">🇫🇷 Français</a>
<a href="README.so.md">🇸🇴 Soomaali</a>
<a href="README.ar.md">🇸🇦 العربية</a>
<a href="README.nl.md">🇧🇪/🇳🇱 Nederlands</a>
<a href="README.hi.md">🇮🇳 हिन्दी</a>
<a href="README.bn.md">🇧🇩 বাংলা</a>
<a href="README.cs.md">🇨🇿 Čeština</a>
<a href="README.ko.md">🇰🇷 한국어</a>
<a href="README.ja.md">🇯🇵 日本語</a>
<!-- Next Flag -->
</p>
<!-- FLAGS:END -->
<!-- NAV:START -->
<p align="center">
<a href="https://www.memvid.com">Website</a>
·
<a href="https://sandbox.memvid.com">Try Sandbox</a>
·
<a href="https://docs.memvid.com">Docs</a>
·
<a href="https://github.com/memvid/memvid/discussions">Discussions</a>
</p>
<!-- NAV:END -->
<!-- BADGES:START -->
<p align="center">
<a href="https://crates.io/crates/memvid-core"><img src="https://img.shields.io/crates/v/memvid-core?style=flat-square&logo=rust" alt="Crates.io" /></a>
<a href="https://docs.rs/memvid-core"><img src="https://img.shields.io/docsrs/memvid-core?style=flat-square&logo=docs.rs" alt="docs.rs" /></a>
<a href="https://github.com/memvid/memvid/blob/main/LICENSE"><img src="https://img.shields.io/badge/license-Apache%202.0-blue?style=flat-square" alt="License" /></a>
</p>
<p align="center">
<a href="https://github.com/memvid/memvid/stargazers"><img src="https://img.shields.io/github/stars/memvid/memvid?style=flat-square&logo=github" alt="Stars" /></a>
<a href="https://github.com/memvid/memvid/network/members"><img src="https://img.shields.io/github/forks/memvid/memvid?style=flat-square&logo=github" alt="Forks" /></a>
<a href="https://github.com/memvid/memvid/issues"><img src="https://img.shields.io/github/issues/memvid/memvid?style=flat-square&logo=github" alt="Issues" /></a>
<a href="https://discord.gg/2mynS7fcK7"><img src="https://img.shields.io/discord/1442910055233224745?style=flat-square&logo=discord&label=discord" alt="Discord" /></a>
</p>
<p align="center">
<a href="https://trendshift.io/repositories/17293" target="_blank"><img src="https://trendshift.io/api/badge/repositories/17293" alt="memvid%2Fmemvid | Trendshift" style="width: 250px; height: 55px;" width="250" height="55"/></a>
</p>
<!-- BADGES:END -->
<p align="center">
<strong>Memvid waa nidaam xusuuseed oo hal fayl ah kaas oo loogu talagalay wakiillada AI (AI agents), lehna soo-celin degdeg ah iyo xusuus fog.</strong><br/>
Xusuus joogto ah, la raadin karo, lana qaadan karo, iyadoo aan loo baahnayn database-yo kale.
</p>
<h2 align="center">⭐️ Noo saar STAR si aad mashruuca u taageerto ⭐️</h2>
## Waa maxay Memvid?
Memvid waa nidaam xusuuseed AI oo la qaadan karo kaas oo kuu keydinaya xogtaada, habka raadinta (embeddings), qaabdhismeedka iyo metadata-daba ku ururiya hal fayl oo keliya.
Halkii aad ka isticmaali lahayd nidaamyada RAG-ga ee adag ama database-yada vector-ka ee ku shaqeeya server-ka, Memvid wuxuu kuu oggolaanayaa inaad xogta si toos ah uga soo ceshato faylka dhexdiisa si aad u degdeg badan.
Natiijadu waa lakab xusuuseed ka madax-bannaan nooca modelka iyo kaabayaasha (infrastructure-free), kaas oo siiya wakiillada AI(AI agents) xusuus joogto ah oo fog oo ay meel walba u qaadan karaan.
---
## Maxay tahay sababta Frames-ka Muuqaalka (Video Frames)?
Memvid wuxuu dhiirrigelin ka helayaa habka xogta muuqaalka loo kaydiyo (video encoding), isaga oo aan kaydinayn muuqaal balse u **habaynaya xusuusta AI si isku-xiga (sequence) oo aad u hufan oo "Smart Frames".**
"Smart Frame" waa unug aan isbeddelayn oo kaydiya macluumaadka oo ay la socdaan waqtiga (timestamps), checksums iyo metadata aasaasi ah. Frames-ka waxaa loo ururiyaa qaab oggolaanaya isku-duubni (compression), tusmeyn (indexing), iyo akhris is-barbar-socda oo hufan.
Qaabdhismeedkan ku salaysan frames-ka wuxuu suuragelinayaa:
- Qoraal kaliya oo lagu darayo (Append-only) iyadoo aan la beddelayn ama la kharribayn xogta jirtay
- Baaritaan lagu sameyn karo xaaladihii xusuusta ee hore
- Kormeeridda habka ay aqoontu u kobcayso iyadoo loo eegayo waqtiga
- Badbaadada xogta (crash safety) iyadoo la adeegsanayo frames go'an oo aan isbeddelayn
- Isku-duubni hufan oo loo adeegsanayo farsamooyin laga soo minguuriyay kaydinta muuqaallada
Natiijadu waa hal fayl oo u dhaqmaya sidii jadwal xusuuseed oo dib loo celin karo oo loogu talagalay nidaamyada AI.
---
## Fikradaha Muhiimka ah
- **Living Memory Engine**
Si joogto ah ugu dar, u laameey (branch), una kobci xusuusta qeybo kala duwan.
- **Capsule Context (`.mv2`)**
Capsule xusuuseed oo isku-filan, la wadaagi karo, lehna sharciyo iyo waqti dhicitaan.
- **Time-Travel Debugging**
Dib u celi, ama qabeey xaalad kasta oo xusuusta ah.
- **Smart Recall**
Soo-celinta xusuusta gudaha wax ka yar 5ms iyadoo la adeegsanayo kaydinta saadaalinta (predictive caching).
- **Codec Intelligence**
Si otomaatig ah u doorta una casriyeeya isku-duubnida (compression) waqtiga ka dib.
---
## Meelaha loo adeegsado (Use Cases)
Memvid waa nidaam xusuuseed oo la qaadan karo oo aan server u baahnayn, kaas oo siiya wakiillada AI(AI agents), xusuus joogto ah iyo soo-celin degdeg ah. Maadaama uu ka madax-bannaan yahay modelka, waxna ku akhriyo qaabab badan (multi-modal), una shaqeeyo si buuxda isagoo aan internet lahayn, horumariyayaashu waxay Memvid u adeegsanayaan hawlo badan:
- Wakiillada AI(AI Agents) ee muddada dheer shaqeeya
- Keydka aqoonta ee shirkadaha
- Tageerida AI-ga ee ku shaqeeya offline-ka
- Fahamka nidaamyada koodhka (Codebase)
- Wakiillada adeegga macmiilka
- Otomaatigga shaqada (Workflow Automation)
- Kaaliyayaasha iibka iyo suuqgeynta
- Kaaliyayaasha aqoonta shakhsi ahaaneed
- Wakiillada caafimaadka, sharciga, iyo maaliyadda
- Hannaanka shaqada AI-ga oo la baari karo lana saxi karo (Auditable/Debuggable)
- Codsiyada gaarka ah (Custom Applications)
---
## SDKs & CLI
Ku isticmaal Memvid luuqadda aad doorbidi lahayd:
| Package | Install | Links |
| --------------- | --------------------------- | ------------------------------------------------------------------------------------------------------------------- |
| **CLI** | `npm install -g memvid-cli` | [![npm](https://img.shields.io/npm/v/memvid-cli?style=flat-square)](https://www.npmjs.com/package/memvid-cli) |
| **Node.js SDK** | `npm install @memvid/sdk` | [![npm](https://img.shields.io/npm/v/@memvid/sdk?style=flat-square)](https://www.npmjs.com/package/@memvid/sdk) |
| **Python SDK** | `pip install memvid-sdk` | [![PyPI](https://img.shields.io/pypi/v/memvid-sdk?style=flat-square)](https://pypi.org/project/memvid-sdk/) |
| **Rust** | `cargo add memvid-core` | [![Crates.io](https://img.shields.io/crates/v/memvid-core?style=flat-square)](https://crates.io/crates/memvid-core) |
---
## Kushubashada (Installation) (Rust)
### Shuruudaha
- **Rust 1.85.0+** — Si aad ugu shubato Guji linkagan [rustup.rs](https://rustup.rs)
### Ku dar Mashruucaaga
```toml
[dependencies]
memvid-core = "2.0"
```
### Feature Flags
| Feature | Sharaxada |
| ------------------- | ---------------------------------------------- |
| `lex` | Raadinta qoraalka oo dhan oo leh darajada BM25 (Tantivy) |
| `pdf_extract` | Soo saarista qoraalka PDF oo saafi ah |
| `vec` | Raadinta isku-midka ah ee Vector (HNSW + ONNX) |
| `clip` | CLIP visual embeddings oo loogu talagalay raadinta sawirka |
| `whisper` | Beddelka codka iyadoo loo baddelayo qoraal lana adeegsanayo Whisper |
| `temporal_track` | Turjumidda taariikhda ee luuqadda caadiga ah ("Salaasadii hore") |
| `parallel_segments` | Soo gelinta xogta iyadoo la adeegsanayo dhowr nuuc (Multi-threaded) |
| `encryption` | capsules-ka xusuusta ee ku xidhan sirta (password) (.mv2e) |
U furo sifooyinka (features) sida aad ugu baahan tahay:
```toml
[dependencies]
memvid-core = { version = "2.0", features = ["lex", "vec", "temporal_track"] }
```
---
## Bilow Degdeg ah (Quick Start)
```rust
use memvid_core::{Memvid, PutOptions, SearchRequest};
fn main() -> memvid_core::Result<()> {
// Create a new memory file
let mut mem = Memvid::create("knowledge.mv2")?;
// Add documents with metadata
let opts = PutOptions::builder()
.title("Meeting Notes")
.uri("mv2://meetings/2024-01-15")
.tag("project", "alpha")
.build();
mem.put_bytes_with_options(b"Q4 planning discussion...", opts)?;
mem.commit()?;
// Search
let response = mem.search(SearchRequest {
query: "planning".into(),
top_k: 10,
snippet_chars: 200,
..Default::default()
})?;
for hit in response.hits {
println!("{}: {}", hit.title.unwrap_or_default(), hit.text);
}
Ok(())
}
```
---
## Dhisid (Build)
Qeyb ka soo qaado (Clone) kaydka (Repository):
```bash
git clone https://github.com/memvid/memvid.git
cd memvid
```
U dhis habka cilad-baadhista (debug mode):
```bash
cargo build
```
U dhis habka rasmiga ah (release mode - la hagaajiyay):
```bash
cargo build --release
```
Ku dhis sifooyin (features) gaar ah:
```bash
cargo build --release --features "lex,vec,temporal_track"
```
---
## Tijaabi iskudayga (Run Tests)
Tijaabi iskudayada oo dhan:
```bash
cargo test
```
Tijaabi iskudayga iyadoo natiijada la arkayo:
```bash
cargo test -- --nocapture
```
Tijaab iskuday gaar ah:
```bash
cargo test test_name
```
Tijaabi iskudayada isku-xirka (integration tests) ah oo keliya
```bash
cargo test --test lifecycle
cargo test --test search
cargo test --test mutation
```
---
## Tusaalooyin (Examples)
Tusaha `examples/` wuxuu ka kooban yahay tusaalooyin shaqaynaya:
### Adeegsiga Fudud
Wuxuu muujinayaa samaynta, gelinta, raadinta, iyo hawlgallada waqtiga (timeline):
```bash
cargo run --example basic_usage
```
### Soo gelinta PDF
Geli oo baadh dukumiintiyada PDF-ka ah (wuxuu isticmaalaa warqaddii aheyd "Attention Is All You Need"):
```bash
cargo run --example pdf_ingestion
```
### Raadinta Muuqaalka ee CLIP
Raadinta sawirka iyadoo la adeegsanayo CLIP (waxay u baahan tahay clip feature):
```bash
cargo run --example clip_visual_search --features clip
```
### Beddelka Codka ee Whisper
Beddelka codka (waxay u baahan tahay whisper feature):
```bash
cargo run --example test_whisper --features whisper
```
---
## Qaabka Faylka (File Format)
Wax walba waxay ku jiraan hal fayl oo .mv2 ah:
```
┌────────────────────────────┐
│ Header (4KB) │ Magic, nooca, awoodda
├────────────────────────────┤
│ Embedded WAL (1-64MB) │ Kasoo kabashada burburka
├────────────────────────────┤
│ Data Segments │ Frames la isku-duubay
├────────────────────────────┤
│ Lex Index │ Tantivy qoraal ka buuxo
├────────────────────────────┤
│ Vec Index │ HNSW vectors
├────────────────────────────┤
│ Time Index │ Siday u kala horreeyaan
├────────────────────────────┤
│ TOC (Footer) │ Meeqaamka qaybaha
└────────────────────────────┘
```
Ma jiraan faylal .wal, .lock, .shm, ama faylal dhinac socda. Weligaa.
Fiiri [MV2_SPEC.md](MV2_SPEC.md) si aad u hesho faahfaahinta dhammaystiran ee qaabka faylka.
---
## Taageer (Support)
Ma qabtaa su'aalo ama ra'yi?
Email: contact@memvid.com
**Noo saar ⭐ si aad u muujiso taageeradaada**
---
## Shatiga (License)
Apache License 2.0 — Fiiri faylka [LICENSE](LICENSE) si aad u hesho faahfaahin dheeraad ah.
+517
View File
@@ -0,0 +1,517 @@
<!-- HEADER:START -->
<img width="2000" height="524" alt="Social Cover (9)"
src="https://github.com/user-attachments/assets/cf66f045-c8be-494b-b696-b8d7e4fb709c" />
<!-- HEADER:END -->
<div style="height: 16px;"></div>
<p align="center">
<a href="https://trendshift.io/repositories/17293" target="_blank"><img src="https://trendshift.io/api/badge/repositories/17293" alt="memvid%2Fmemvid | Trendshift" style="width: 250px; height: 55px;" width="250" height="55"/></a>
</p>
<!-- BADGES:END -->
<p align="center">
<strong>Memvid 是专为 AI 智能体设计的单文件记忆层,具备即时检索和长期记忆能力。</strong><br/>
持久化、版本化、可移植的记忆,无需数据库。
</p>
<!-- NAV:START -->
<p align="center">
<a href="https://www.memvid.com">官方网站</a>
·
<a href="https://sandbox.memvid.com">尝试一下沙箱</a>
·
<a href="https://docs.memvid.com">文档</a>
·
<a href="https://github.com/memvid/memvid/discussions">讨论区</a>
</p>
<!-- NAV:END -->
<!-- BADGES:START -->
<p align="center">
<a href="https://crates.io/crates/memvid-core"><img src="https://img.shields.io/crates/v/memvid-core?style=flat-square&logo=rust" alt="Crates.io" /></a>
<a href="https://docs.rs/memvid-core"><img src="https://img.shields.io/docsrs/memvid-core?style=flat-square&logo=docs.rs" alt="docs.rs" /></a>
<a href="https://github.com/memvid/memvid/blob/main/LICENSE"><img src="https://img.shields.io/badge/license-Apache%202.0-blue?style=flat-square" alt="许可证" /></a>
</p>
<p align="center">
<a href="https://github.com/memvid/memvid/stargazers"><img src="https://img.shields.io/github/stars/memvid/memvid?style=flat-square&logo=github" alt="Stars" /></a>
<a href="https://github.com/memvid/memvid/network/members"><img src="https://img.shields.io/github/forks/memvid/memvid?style=flat-square&logo=github" alt="Forks" /></a>
<a href="https://github.com/memvid/memvid/issues"><img src="https://img.shields.io/github/issues/memvid/memvid?style=flat-square&logo=github" alt="Issues" /></a>
<a href="https://discord.gg/2mynS7fcK7"><img src="https://img.shields.io/discord/1442910055233224745?style=flat-square&logo=discord&label=discord" alt="Discord" /></a>
</p>
<h2 align="center">⭐️ 给项目点个星标支持我们 ⭐️</h2>
## 基准测试亮点
**🚀 准确率超越其他记忆系统:** 在 LoCoMo 上领先 SOTA 35%,长时对话回忆与推理能力最佳
**🧠 卓越的多跳与时序推理:** 比行业平均水平高出 76% 多跳推理,56% 时序推理
**⚡ 超低延迟高吞吐:** P50 仅 0.025msP99 仅 0.075ms,吞吐量是标准的 1,372 倍
**🔬 完全可复现的基准测试:** LoCoMo10 次约 26K token 的对话)、开源评估、LLM-as-Judge
## 什么是 Memvid
Memvid 是可移植的 AI 记忆系统,将数据、嵌入向量、搜索结构和元数据打包成单个文件。
无需运行复杂的 RAG 管道或基于服务器的向量数据库,Memvid 支持直接从文件进行快速检索。
结果是模型无关、无基础设施的记忆层,让 AI 智能体拥有可随身携带的持久化长期记忆。
## 什么是 Smart Frames
Memvid 借鉴视频编码的理念,不是为了存储视频,而是**将 AI 记忆组织为仅追加、超高效序列的 Smart Frames。**
Smart Frame 是存储内容以及时间戳、校验和和基本元数据的不可变单元。
帧以允许高效压缩、索引和并行读取的方式分组。
这种基于帧的设计支持:
- 仅追加写入,不修改或破坏现有数据
- 对过去记忆状态的查询
- 知识演化的时间轴式检查
- 通过基于提交的不可变帧应对崩溃
- 使用基于视频编码技术的高效压缩
结果是一个表现为 AI 系统可追溯记忆时间线的单文件。
## 核心概念
- **Living Memory Engine**
持续追加、分支和跨会话演进记忆。
- **Capsule Context (`.mv2`)**
自包含、可共享的记忆胶囊,带规则和过期时间。
- **Time-Travel Debugging**
回溯、重放或分支化任何记忆状态。
- **Smart Recall**
小于 5ms 本地记忆访问,具备预测性缓存。
- **Codec Intelligence**
随时间自动选择和升级压缩。
## 使用场景
Memvid 是无服务器便携记忆层,为 AI 智能体提供持久记忆和快速召回。由于它是模型无关、多模态且完全离线工作,开发者正在各种现实应用中广泛的使用 Memvid。
- 长期运行的 AI 智能体
- 企业知识库
- 离线优先 AI 系统
- 代码库理解
- 客户支持智能体
- 工作流自动化
- 销售与营销助手
- 个人知识助理
- 医疗、法律和金融智能体
- 可审计和可调试的 AI 工作流
- 自定义应用
## SDKs 与 CLI
在您喜欢的语言中使用 Memvid
| 包 | 安装 | 链接 |
| --------------- | --------------------------- | ------------------------------------------------------------------------------------------------------------------- |
| **CLI** | `npm install -g memvid-cli` | [![npm](https://img.shields.io/npm/v/memvid-cli?style=flat-square)](https://www.npmjs.com/package/memvid-cli) |
| **Node.js SDK** | `npm install @memvid/sdk` | [![npm](https://img.shields.io/npm/v/@memvid/sdk?style=flat-square)](https://www.npmjs.com/package/@memvid/sdk) |
| **Python SDK** | `pip install memvid-sdk` | [![PyPI](https://img.shields.io/pypi/v/memvid-sdk?style=flat-square)](https://pypi.org/project/memvid-sdk/) |
| **Rust** | `cargo add memvid-core` | [![Crates.io](https://img.shields.io/crates/v/memvid-core?style=flat-square)](https://crates.io/crates/memvid-core) |
---
## 安装(Rust
### 要求
- **Rust 1.85.0+** — 从 [rustup.rs](https://rustup.rs) 安装
### 添加到项目
```toml
[dependencies]
memvid-core = "2.0"
```
### 功能标志
| 功能 | 描述 |
| -------------------- | ---------------------------------------------------------------- |
| `lex` | 使用 BM25 排序的全文搜索(Tantivy |
| `pdf_extract` | 纯 Rust PDF 文本提取 |
| `vec` | 向量相似搜索(HNSW + 通过 ONNX 的本地文本嵌入) |
| `clip` | CLIP 视觉嵌入用于图像搜索 |
| `whisper` | 使用 Whisper 进行音频转录 |
| `api_embed` | 云 API 嵌入(OpenAI |
| `temporal_track` | 自然语言日期解析("last Tuesday" |
| `parallel_segments` | 多线程摄取 |
| `encryption` | 基于密码的加密胶囊(.mv2e) |
| `symspell_cleanup` | 强大的 PDF 文本修复(修复 "emp lo yee" -> "employee" |
按需启用功能:
```toml
[dependencies]
memvid-core = { version = "2.0", features = ["lex", "vec", "temporal_track"] }
```
## 快速开始
```rust
use memvid_core::{Memvid, PutOptions, SearchRequest};
fn main() -> memvid_core::Result<()> {
// 创建新记忆文件
let mut mem = Memvid::create("knowledge.mv2")?;
// 添加带元数据的文档
let opts = PutOptions::builder()
.title("Meeting Notes")
.uri("mv2://meetings/2024-01-15")
.tag("project", "alpha")
.build();
mem.put_bytes_with_options(b"Q4 planning discussion...", opts)?;
mem.commit()?;
// 搜索
let response = mem.search(SearchRequest {
query: "planning".into(),
top_k: 10,
snippet_chars: 200,
..Default::default()
})?;
for hit in response.hits {
println!("{}: {}", hit.title.unwrap_or_default(), hit.text);
}
Ok(())
}
```
---
## 构建
克隆仓库:
```bash
git clone https://github.com/memvid/memvid.git
cd memvid
```
以调试模式构建:
```bash
cargo build
```
以发布模式构建(优化):
```bash
cargo build --release
```
使用特定功能构建:
```bash
cargo build --release --features "lex,vec,temporal_track"
```
---
## 运行测试
运行所有测试:
```bash
cargo test
```
带输出运行测试:
```bash
cargo test -- --nocapture
```
运行特定测试:
```bash
cargo test test_name
```
仅运行集成测试:
```bash
cargo test --test lifecycle
cargo test --test search
cargo test --test mutation
```
---
## 示例
`examples/` 目录包含可运行示例:
### 基本用法
演示创建、添加、搜索和时间线操作:
```bash
cargo run --example basic_usage
```
### PDF 提取
提取和搜索 PDF 文档(使用 "Attention Is All You Need" 论文):
```bash
cargo run --example pdf_ingestion
```
### CLIP 可视化搜索
使用 CLIP 嵌入进行图像搜索(需要 `clip` 功能):
```bash
cargo run --example clip_visual_search --features clip
```
### Whisper 转录
音频转录(需要 `whisper` 功能):
```bash
cargo run --example test_whisper --features whisper -- /path/to/audio.mp3
```
**可用模型:**
| 模型 | 大小 | 速度 | 用例 |
| ---------------------- | ------ | ------- | ----------------------------------- |
| `whisper-small-en` | 244 MB | 最慢 | 最佳准确度(默认) |
| `whisper-tiny-en` | 75 MB | 快 | 平衡 |
| `whisper-tiny-en-q8k` | 19 MB | 最快 | 快速测试,资源受限 |
**模型选择:**
```bash
# 默认(FP32 small,最高准确度)
cargo run --example test_whisper --features whisper -- audio.mp3
# 小型量化(小 75%,更快)
MEMVID_WHISPER_MODEL=whisper-tiny-en-q8k cargo run --example test_whisper --features whisper -- audio.mp3
```
**可编程配置:**
```rust
use memvid_core::{WhisperConfig, WhisperTranscriber};
// 默认 FP32 small 模型
let config = WhisperConfig::default();
// 小型量化模型(更快,更小)
let config = WhisperConfig::with_quantization();
// 特定模型
let config = WhisperConfig::with_model("whisper-tiny-en-q8k");
let transcriber = WhisperTranscriber::new(&config)?;
let result = transcriber.transcribe_file("audio.mp3")?;
println!("{}", result.text);
```
## 文本嵌入模型
`vec` 功能包括使用 ONNX 模型的本地文本嵌入支持。在使用本地文本嵌入之前,您需要手动下载模型文件。
### 快速开始:BGE-small(推荐)
下载默认 BGE-small 模型(384 维,快速高效):
```bash
mkdir -p ~/.cache/memvid/text-models
# 下载 ONNX 模型
curl -L 'https://huggingface.co/BAAI/bge-small-en-v1.5/resolve/main/onnx/model.onnx' \
-o ~/.cache/memvid/text-models/bge-small-en-v1.5.onnx
# 下载分词器
curl -L 'https://huggingface.co/BAAI/bge-small-en-v1.5/resolve/main/tokenizer.json' \
-o ~/.cache/memvid/text-models/bge-small-en-v1.5_tokenizer.json
```
### 可用模型
| 模型 | 维度 | 大小 | 最适合 |
| ------------------------ | ---------- | ------ | --------------- |
| `bge-small-en-v1.5` | 384 | ~120MB | 默认,快速 |
| `bge-base-en-v1.5` | 768 | ~420MB | 更好的质量 |
| `nomic-embed-text-v1.5` | 768 | ~530MB | 多用途任务 |
| `gte-large` | 1024 | ~1.3GB | 最高质量 |
### 其他模型
**BGE-base**768 维):
```bash
curl -L 'https://huggingface.co/BAAI/bge-base-en-v1.5/resolve/main/onnx/model.onnx' \
-o ~/.cache/memvid/text-models/bge-base-en-v1.5.onnx
curl -L 'https://huggingface.co/BAAI/bge-base-en-v1.5/resolve/main/tokenizer.json' \
-o ~/.cache/memvid/text-models/bge-base-en-v1.5_tokenizer.json
```
**Nomic**768 维):
```bash
curl -L 'https://huggingface.co/nomic-ai/nomic-embed-text-v1.5/resolve/main/onnx/model.onnx' \
-o ~/.cache/memvid/text-models/nomic-embed-text-v1.5.onnx
curl -L 'https://huggingface.co/nomic-ai/nomic-embed-text-v1.5/resolve/main/tokenizer.json' \
-o ~/.cache/memvid/text-models/nomic-embed-text-v1.5_tokenizer.json
```
**GTE-large**1024 维):
```bash
curl -L 'https://huggingface.co/thenlper/gte-large/resolve/main/onnx/model.onnx' \
-o ~/.cache/memvid/text-models/gte-large.onnx
curl -L 'https://huggingface.co/thenlper/gte-large/resolve/main/tokenizer.json' \
-o ~/.cache/memvid/text-models/gte-large_tokenizer.json
```
### 在代码中使用
```rust
use memvid_core::text_embed::{LocalTextEmbedder, TextEmbedConfig};
use memvid_core::types::embedding::EmbeddingProvider;
// 使用默认模型(BGE-small
let config = TextEmbedConfig::default();
let embedder = LocalTextEmbedder::new(config)?;
let embedding = embedder.embed_text("hello world")?;
assert_eq!(embedding.len(), 384);
// 使用不同模型
let config = TextEmbedConfig::bge_base();
let embedder = LocalTextEmbedder::new(config)?;
```
有关相似度计算和搜索排名的完整示例,请参见 `examples/text_embedding.rs`
### 模型一致性
为防止意外地模型混合(例如,使用 OpenAI 嵌入查询 BGE-small 索引),您可以将 Memvid 实例显式绑定到特定模型名称:
```rust
// 将索引绑定到特定模型。
// 如果之前使用不同模型创建索引将返回错误。
mem.set_vec_model("bge-small-en-v1.5")?;
```
绑定是持久化的。一旦设置,将来尝试使用不同模型名称将快速失败并返回 `ModelMismatch` 错误。
## API 嵌入(OpenAI
`api_embed` 功能使用 OpenAI 的 API 启用基于云的嵌入生成。
### 设置
设置您的 OpenAI API 密钥:
```bash
export OPENAI_API_KEY="sk-..."
```
### 用法
```rust
use memvid_core::api_embed::{OpenAIConfig, OpenAIEmbedder};
use memvid_core::types::embedding::EmbeddingProvider;
// 使用默认模型(text-embedding-3-small
let config = OpenAIConfig::default();
let embedder = OpenAIEmbedder::new(config)?;
let embedding = embedder.embed_text("hello world")?;
assert_eq!(embedding.len(), 1536);
// 使用更高质量模型
let config = OpenAIConfig::large(); // text-embedding-3-large (3072 维)
let embedder = OpenAIEmbedder::new(config)?;
```
### 可用模型
| 模型 | 维度 | 最适合 |
| ------------------------ | ---------- | -------------------------- |
| `text-embedding-3-small` | 1536 | 默认,最快,最便宜 |
| `text-embedding-3-large` | 3072 | 最高质量 |
| `text-embedding-ada-002` | 1536 | 传统模型 |
有关完整示例,请参见 `examples/openai_embedding.rs`
## 文件格式
所有内容都存储在单个 `.mv2` 文件中:
```
┌────────────────────────────┐
│ Header (4KB) │ 魔数,版本,容量
├────────────────────────────┤
│ Embedded WAL (1-64MB) │ 崩溃恢复
├────────────────────────────┤
│ Data Segments │ 压缩帧
├────────────────────────────┤
│ Lex Index │ Tantivy 全文
├────────────────────────────┤
│ Vec Index │ HNSW 向量
├────────────────────────────┤
│ Time Index │ 时序排序
├────────────────────────────┤
│ TOC (Footer) │ 段偏移
└────────────────────────────┘
```
不会有 `.wal``.lock``.shm` 或附带文件。永远不会。
有关完整文件格式规范,请参见 [MV2_SPEC.md](MV2_SPEC.md)。
## 支持
有问题或反馈?
邮箱:contact@memvid.com
**点个 ⭐ 支持我们**
---
> **Memvid v1(基于 QR 码的记忆)已弃用**
>
> 如果您参考的是 QR 码,那么您正在使用过时信息。
>
> 参见:https://docs.memvid.com/memvid-v1-deprecation
---
## 许可证
Apache License 2.0 — 详细信息请参见 [LICENSE](LICENSE) 文件。
+249
View File
@@ -0,0 +1,249 @@
const fs = require('fs');
const path = require('path');
const I18N_DIR = path.join(__dirname, '..');
const ROOT_DIR = path.join(I18N_DIR, '..', '..');
const README_PATH = path.join(ROOT_DIR, 'README.md');
const LANG_MAP = {
'aa': { emoji: '🌐', name: 'Afar' },
'ab': { emoji: '🌐', name: 'Abkhazian' },
'ae': { emoji: '🌐', name: 'Avestan' },
'af': { emoji: '🇿🇦', name: 'Afrikaans' },
'ak': { emoji: '🌐', name: 'Akan' },
'am': { emoji: '🇪🇹', name: 'Amharic' },
'an': { emoji: '🌐', name: 'Aragonese' },
'ar': { emoji: '🇸🇦', name: 'العربية' },
'as': { emoji: '🌐', name: 'Assamese' },
'av': { emoji: '🌐', name: 'Avaric' },
'ay': { emoji: '🌐', name: 'Aymara' },
'az': { emoji: '🇦🇿', name: 'Azerbaijani' },
'ba': { emoji: '🌐', name: 'Bashkir' },
'be': { emoji: '🇧🇾', name: 'Belarusian' },
'bg': { emoji: '🇧🇬', name: 'Bulgarian' },
'bi': { emoji: '🌐', name: 'Bislama' },
'bm': { emoji: '🌐', name: 'Bambara' },
'bn': { emoji: '🇧🇩', name: 'বাংলা' },
'bo': { emoji: '🌐', name: 'Tibetan' },
'br': { emoji: '🌐', name: 'Breton' },
'bs': { emoji: '🇧🇦', name: 'Bosnian' },
'ca': { emoji: '🇪🇸', name: 'Catalan' },
'ce': { emoji: '🌐', name: 'Chechen' },
'ch': { emoji: '🌐', name: 'Chamorro' },
'co': { emoji: '🌐', name: 'Corsican' },
'cr': { emoji: '🌐', name: 'Cree' },
'cs': { emoji: '🇨🇿', name: 'Česko' },
'cu': { emoji: '🌐', name: 'Church Slavonic' },
'cv': { emoji: '🌐', name: 'Chuvash' },
'cy': { emoji: '🇬🇧', name: 'Welsh' },
'da': { emoji: '🇩🇰', name: 'Danish' },
'de': { emoji: '🇩🇪', name: 'Deutsch' },
'dv': { emoji: '🌐', name: 'Divehi' },
'dz': { emoji: '🇧🇹', name: 'Dzongkha' },
'ee': { emoji: '🌐', name: 'Ewe' },
'el': { emoji: '🇬🇷', name: 'Greek' },
'en': { emoji: '🇺🇸', name: 'English' },
'eo': { emoji: '🌐', name: 'Esperanto' },
'es': { emoji: '🇪🇸', name: 'Español' },
'et': { emoji: '🇪🇪', name: 'Estonian' },
'eu': { emoji: '🌐', name: 'Basque' },
'fa': { emoji: '🇮🇷', name: 'Persian' },
'ff': { emoji: '🌐', name: 'Fulah' },
'fi': { emoji: '🇫🇮', name: 'Finnish' },
'fj': { emoji: '🌐', name: 'Fijian' },
'fo': { emoji: '🌐', name: 'Faroese' },
'fr': { emoji: '🇫🇷', name: 'Français' },
'fy': { emoji: '🌐', name: 'Western Frisian' },
'ga': { emoji: '🇮🇪', name: 'Irish' },
'gd': { emoji: '🌐', name: 'Gaelic' },
'gl': { emoji: '🌐', name: 'Galician' },
'gn': { emoji: '🌐', name: 'Guarani' },
'gu': { emoji: '🌐', name: 'Gujarati' },
'gv': { emoji: '🌐', name: 'Manx' },
'ha': { emoji: '🇳🇬', name: 'Hausa' },
'he': { emoji: '🇮🇱', name: 'Hebrew' },
'hi': { emoji: '🇮🇳', name: 'हिन्दी' },
'ho': { emoji: '🌐', name: 'Hiri Motu' },
'hr': { emoji: '🇭🇷', name: 'Croatian' },
'ht': { emoji: '🌐', name: 'Haitian' },
'hu': { emoji: '🇭🇺', name: 'Hungarian' },
'hy': { emoji: '🇦🇲', name: 'Armenian' },
'hz': { emoji: '🌐', name: 'Herero' },
'ia': { emoji: '🌐', name: 'Interlingua' },
'id': { emoji: '🇮🇩', name: 'Bahasa' },
'ie': { emoji: '🌐', name: 'Interlingue' },
'ig': { emoji: '🇳🇬', name: 'Igbo' },
'ii': { emoji: '🌐', name: 'Sichuan Yi' },
'ik': { emoji: '🌐', name: 'Inupiaq' },
'io': { emoji: '🌐', name: 'Ido' },
'is': { emoji: '🇮🇸', name: 'Icelandic' },
'it': { emoji: '🇮🇹', name: 'Italiano' },
'iu': { emoji: '🌐', name: 'Inuktitut' },
'ja': { emoji: '🇯🇵', name: '日本語' },
'jv': { emoji: '🌐', name: 'Javanese' },
'ka': { emoji: '🇬🇪', name: 'Georgian' },
'kg': { emoji: '🌐', name: 'Kongo' },
'ki': { emoji: '🌐', name: 'Kikuyu' },
'kj': { emoji: '🌐', name: 'Kuanyama' },
'kk': { emoji: '🇰🇿', name: 'Kazakh' },
'kl': { emoji: '🌐', name: 'Kalaallisut' },
'km': { emoji: '🇰🇭', name: 'Central Khmer' },
'kn': { emoji: '🌐', name: 'Kannada' },
'ko': { emoji: '🇰🇷', name: '한국어' },
'kr': { emoji: '🌐', name: 'Kanuri' },
'ks': { emoji: '🌐', name: 'Kashmiri' },
'ku': { emoji: '🇮🇶', name: 'Kurdish' },
'kv': { emoji: '🌐', name: 'Komi' },
'kw': { emoji: '🌐', name: 'Cornish' },
'ky': { emoji: '🇰🇬', name: 'Kyrgyz' },
'la': { emoji: '🌐', name: 'Latin' },
'lb': { emoji: '🌐', name: 'Luxembourgish' },
'lg': { emoji: '🌐', name: 'Ganda' },
'li': { emoji: '🌐', name: 'Limburgan' },
'ln': { emoji: '🌐', name: 'Lingala' },
'lo': { emoji: '🇱🇦', name: 'Lao' },
'lt': { emoji: '🇱🇹', name: 'Lithuanian' },
'lu': { emoji: '🌐', name: 'Luba-Katanga' },
'lv': { emoji: '🇱🇻', name: 'Latvian' },
'mg': { emoji: '🌐', name: 'Malagasy' },
'mh': { emoji: '🌐', name: 'Marshallese' },
'mi': { emoji: '🌐', name: 'Maori' },
'mk': { emoji: '🇲🇰', name: 'Macedonian' },
'ml': { emoji: '🌐', name: 'Malayalam' },
'mn': { emoji: '🇲🇳', name: 'Mongolian' },
'mr': { emoji: '🌐', name: 'Marathi' },
'ms': { emoji: '🇲🇾', name: 'Malay' },
'mt': { emoji: '🇲🇹', name: 'Maltese' },
'my': { emoji: '🇲🇲', name: 'Burmese' },
'na': { emoji: '🌐', name: 'Nauru' },
'nb': { emoji: '🌐', name: 'Norwegian Bokmål' },
'nd': { emoji: '🌐', name: 'North Ndebele' },
'ne': { emoji: '🇳🇵', name: 'Nepali' },
'ng': { emoji: '🌐', name: 'Ndonga' },
'nl': { emoji: '🇧🇪/🇳🇱', name: 'Nederlands' },
'nn': { emoji: '🌐', name: 'Norwegian Nynorsk' },
'no': { emoji: '🇳🇴', name: 'Norwegian' },
'nr': { emoji: '🌐', name: 'South Ndebele' },
'nv': { emoji: '🌐', name: 'Navajo' },
'ny': { emoji: '🌐', name: 'Chichewa' },
'oc': { emoji: '🌐', name: 'Occitan' },
'oj': { emoji: '🌐', name: 'Ojibwa' },
'om': { emoji: '🌐', name: 'Oromo' },
'or': { emoji: '🌐', name: 'Oriya' },
'os': { emoji: '🌐', name: 'Ossetian' },
'pa': { emoji: '🌐', name: 'Punjabi' },
'pi': { emoji: '🌐', name: 'Pali' },
'pl': { emoji: '🇵', name: 'Polski' },
'ps': { emoji: '🇦🇫', name: 'Pashto' },
'pt': { emoji: '🇵🇹', name: 'Português' },
'qu': { emoji: '🌐', name: 'Quechua' },
'rm': { emoji: '🌐', name: 'Romansh' },
'rn': { emoji: '🌐', name: 'Rundi' },
'ro': { emoji: '🇷🇴', name: 'Romanian' },
'ru': { emoji: '🇷🇺', name: 'Русский' },
'rw': { emoji: '🌐', name: 'Kinyarwanda' },
'sa': { emoji: '🌐', name: 'Sanskrit' },
'sc': { emoji: '🌐', name: 'Sardinian' },
'sd': { emoji: '🌐', name: 'Sindhi' },
'se': { emoji: '🌐', name: 'Northern Sami' },
'sg': { emoji: '🌐', name: 'Sango' },
'si': { emoji: '🇱🇰', name: 'Sinhala' },
'sk': { emoji: '🇸🇰', name: 'Slovak' },
'sl': { emoji: '🇸🇮', name: 'Slovenian' },
'sm': { emoji: '🌐', name: 'Samoan' },
'sn': { emoji: '🌐', name: 'Shona' },
'so': { emoji: '🇸🇴', name: 'Soomaali' },
'sq': { emoji: '🇦🇱', name: 'Albanian' },
'sr': { emoji: '🇷🇸', name: 'Serbian' },
'ss': { emoji: '🌐', name: 'Swati' },
'st': { emoji: '🇿🇦', name: 'Southern Sotho' },
'su': { emoji: '🌐', name: 'Sundanese' },
'sv': { emoji: '🇸🇪', name: 'Swedish' },
'sw': { emoji: '🇰🇪', name: 'Swahili' },
'ta': { emoji: '🌐', name: 'Tamil' },
'te': { emoji: '🌐', name: 'Telugu' },
'tg': { emoji: '🇹🇯', name: 'Tajik' },
'th': { emoji: '🇹🇭', name: 'Thai' },
'ti': { emoji: '🌐', name: 'Tigrinya' },
'tk': { emoji: '🇹🇲', name: 'Turkmen' },
'tl': { emoji: '🇵🇭', name: 'Tagalog' },
'tn': { emoji: '🌐', name: 'Tswana' },
'to': { emoji: '🌐', name: 'Tonga' },
'tr': { emoji: '🇹🇷', name: 'Türkçe' },
'ts': { emoji: '🌐', name: 'Tsonga' },
'tt': { emoji: '🌐', name: 'Tatar' },
'tw': { emoji: '🌐', name: 'Twi' },
'ty': { emoji: '🌐', name: 'Tahitian' },
'ug': { emoji: '🌐', name: 'Uighur' },
'uk': { emoji: '🇺🇦', name: 'Ukrainian' },
'ur': { emoji: '🇵🇰', name: 'Urdu' },
'uz': { emoji: '🇺🇿', name: 'Uzbek' },
've': { emoji: '🌐', name: 'Venda' },
'vi': { emoji: '🇻🇳', name: 'Tiếng Việt' },
'vo': { emoji: '🌐', name: 'Volapük' },
'wa': { emoji: '🌐', name: 'Walloon' },
'wo': { emoji: '🌐', name: 'Wolof' },
'xh': { emoji: '🇿🇦', name: 'Xhosa' },
'yi': { emoji: '🌐', name: 'Yiddish' },
'yo': { emoji: '🇳🇬', name: 'Yoruba' },
'za': { emoji: '🌐', name: 'Zhuang' },
'zh': { emoji: '🇨🇳', name: '中文' },
'zh-CN': { emoji: '🇨🇳', name: '中文 (简体)' },
'zh-HK': { emoji: '🇭🇰', name: '中文 (繁體)' },
'zh-Hans': { emoji: '🇨🇳', name: '中文 (简体)' },
'zh-Hant': { emoji: '🇹🇼', name: '中文 (繁體)' },
'zh-MO': { emoji: '🇲🇴', name: '中文 (繁體)' },
'zh-SG': { emoji: '🇸🇬', name: '中文 (繁體)' },
'zh-TW': { emoji: '🇹🇼', name: '中文 (繁體)' },
'zu': { emoji: '🇿🇦', name: 'Zulu' },
};
function autoAddFlags() {
if (!fs.existsSync(README_PATH)) {
console.error('Error: Cannot find ' + README_PATH);
process.exit(1);
}
let readmeContent = fs.readFileSync(README_PATH, 'utf-8');
const marker = ' <!-- Next Flag -->';
if (!readmeContent.includes(marker)) {
console.warn('Error: <!-- Next Flag --> marker not found in ' + README_PATH);
console.warn('Please add the marker where you want new flags to be inserted.');
return;
}
const files = fs.readdirSync(I18N_DIR);
const translationFiles = files.filter(f =>
f.startsWith('README') &&
f.endsWith('.md') &&
f !== 'README.md'
);
let updated = false;
translationFiles.forEach(file => {
const code = file.split('.')[1];
const lang = LANG_MAP[code];
if (!lang) return;
const flagLink = ` <a href="docs/i18n/${file}">${lang.emoji} ${lang.name}</a>`;
if (!readmeContent.includes(file)) {
readmeContent = readmeContent.replace(marker, flagLink + '\n' + marker);
updated = true;
console.log('Added ' + code);
}
});
if (updated) {
fs.writeFileSync(README_PATH, readmeContent, 'utf-8');
console.log('Main README updated.');
} else {
console.log('No new flags to add.');
}
}
autoAddFlags();
@@ -0,0 +1,114 @@
const fs = require('fs');
const path = require('path');
const I18N_DIR = path.join(__dirname, '..');
const ROOT_DIR = path.join(I18N_DIR, '..', '..');
const README_PATH = path.join(ROOT_DIR, 'README.md');
const MARKERS = {
HEADER: {
START: '<!-- HEADER:START -->',
END: '<!-- HEADER:END -->'
},
FLAGS: {
START: '<!-- FLAGS:START -->',
END: '<!-- FLAGS:END -->'
},
NAV: {
START: '<!-- NAV:START -->',
END: '<!-- NAV:END -->'
},
BADGES: {
START: '<!-- BADGES:START -->',
END: '<!-- BADGES:END -->'
}
};
function extractBlock(content, marker) {
const startIdx = content.indexOf(marker.START);
const endIdx = content.indexOf(marker.END);
if (startIdx === -1 || endIdx === -1) {
return null;
}
return content.substring(startIdx, endIdx + marker.END.length);
}
function updateOrInsertBlock(targetContent, blockKey, blockValue, fallbackAnchor = null) {
const marker = MARKERS[blockKey];
const startIdx = targetContent.indexOf(marker.START);
const endIdx = targetContent.indexOf(marker.END);
if (startIdx !== -1 && endIdx !== -1) {
const prefix = targetContent.substring(0, startIdx);
const suffix = targetContent.substring(endIdx + marker.END.length);
return prefix + blockValue + suffix;
}
if (startIdx !== -1 || endIdx !== -1) {
let cleaned = targetContent.split(marker.START).join('');
cleaned = cleaned.split(marker.END).join('');
return updateOrInsertBlock(cleaned, blockKey, blockValue, fallbackAnchor);
}
if (fallbackAnchor) {
const anchorIdx = targetContent.indexOf(fallbackAnchor);
if (anchorIdx !== -1) {
const insertAfterIdx = targetContent.indexOf('\n', anchorIdx) + 1;
const prefix = targetContent.substring(0, insertAfterIdx);
const suffix = targetContent.substring(insertAfterIdx);
return prefix + '\n' + blockValue + '\n' + suffix;
}
}
return targetContent + '\n\n' + blockValue;
}
function updateLocalizedReadmes() {
if (!fs.existsSync(README_PATH)) {
console.error('Error: Cannot find ' + README_PATH);
process.exit(1);
}
const readmeContent = fs.readFileSync(README_PATH, 'utf-8');
const headerBlock = extractBlock(readmeContent, MARKERS.HEADER);
const flagsBlock = extractBlock(readmeContent, MARKERS.FLAGS);
const navBlock = extractBlock(readmeContent, MARKERS.NAV);
const badgesBlock = extractBlock(readmeContent, MARKERS.BADGES);
if (!headerBlock || !flagsBlock || !navBlock || !badgesBlock) {
console.error('Error: Some markers are missing in main README.md');
if (!headerBlock) console.error('Missing: HEADER');
if (!flagsBlock) console.error('Missing: FLAGS');
if (!navBlock) console.error('Missing: NAV');
if (!badgesBlock) console.error('Missing: BADGES');
process.exit(1);
}
let flagsI18n = flagsBlock.split('href="docs/i18n/').join('href="');
flagsI18n = flagsI18n.split('href="README.md"').join('href="../../README.md"');
const files = fs.readdirSync(I18N_DIR);
const targetFiles = files
.filter(file => file.startsWith('README.') && file.endsWith('.md') && file !== 'README.md')
.map(file => path.join(I18N_DIR, file));
targetFiles.forEach(filePath => {
let content = fs.readFileSync(filePath, 'utf-8');
const fileName = path.basename(filePath);
content = updateOrInsertBlock(content, 'HEADER', headerBlock, '<img');
content = updateOrInsertBlock(content, 'FLAGS', flagsI18n, MARKERS.HEADER.END);
content = updateOrInsertBlock(content, 'NAV', navBlock, MARKERS.FLAGS.END);
content = updateOrInsertBlock(content, 'BADGES', badgesBlock, MARKERS.NAV.END);
fs.writeFileSync(filePath, content, 'utf-8');
console.log('Updated ' + fileName);
});
console.log('\nSuccess: Done.');
}
updateLocalizedReadmes();
+4
View File
@@ -97,6 +97,8 @@ fn main() -> Result<()> {
as_of_frame: None,
as_of_ts: None,
no_sketch: false,
acl_context: None,
acl_enforcement_mode: memvid_core::types::AclEnforcementMode::Audit,
};
let response = mem.search(request)?;
println!(" Query: 'memvid'");
@@ -126,6 +128,8 @@ fn main() -> Result<()> {
as_of_frame: None,
as_of_ts: None,
no_sketch: false,
acl_context: None,
acl_enforcement_mode: memvid_core::types::AclEnforcementMode::Audit,
};
let response = mem.search(request)?;
println!(" Query: 'documentation' (scope: mv2://docs/)");
+1 -1
View File
@@ -27,7 +27,7 @@ fn main() -> memvid_core::Result<()> {
{
eprintln!("This example requires the 'clip' feature.");
eprintln!("Run with: cargo run --example clip_visual_search --features clip");
return Ok(());
Ok(())
}
#[cfg(feature = "clip")]
+162
View File
@@ -0,0 +1,162 @@
//! Generate visual performance comparison report
//!
//! Run with: cargo run --example generate_performance_report --features lex
use memvid_core::{Memvid, PutOptions, SearchRequest};
use std::time::Instant;
fn main() -> memvid_core::Result<()> {
println!("=== Search Precision Performance Report ===\n");
// Create test corpus
println!("Setting up test corpus (1000 documents)...");
let temp_file = "/tmp/perf_test.mv2";
let _ = std::fs::remove_file(temp_file);
let mut mem = Memvid::create(temp_file)?;
// Add 1000 documents with controlled distribution
for i in 0..1000 {
let topic = match i % 5 {
0 => ("machine learning", "neural networks"),
1 => ("python programming", "software development"),
2 => ("machine learning with python", "data science"),
3 => ("rust systems programming", "memory safety"),
_ => ("web development", "javascript frameworks"),
};
let content = format!(
"Document {} about {}. This article covers {} in depth.",
i, topic.0, topic.1
);
mem.put_bytes_with_options(
content.as_bytes(),
PutOptions::builder()
.title(format!("Doc {} - {}", i, topic.0))
.build(),
)?;
if (i + 1) % 100 == 0 {
mem.commit()?;
}
}
mem.commit()?;
println!("✓ Corpus ready\n");
// Test queries
let test_queries = vec![
("machine python", "Both terms"),
("machine learning python", "Three terms"),
("python programming development", "Three terms"),
("rust memory safety", "Two terms"),
];
println!("┌─────────────────────────────────────────────────────────────────────┐");
println!("│ QUERY PERFORMANCE METRICS │");
println!("├─────────────────────────────────────────────────────────────────────┤");
for (query, desc) in &test_queries {
// Warm up
for _ in 0..10 {
let _ = mem.search(SearchRequest {
query: query.to_string(),
top_k: 100,
snippet_chars: 200,
uri: None,
scope: None,
cursor: None,
#[cfg(feature = "temporal_track")]
temporal: None,
as_of_frame: None,
as_of_ts: None,
no_sketch: false,
acl_context: None,
acl_enforcement_mode: memvid_core::types::AclEnforcementMode::Audit,
})?;
}
// Measure
let iterations = 100;
let start = Instant::now();
let mut total_results = 0;
let mut total_relevant = 0;
for _ in 0..iterations {
let results = mem.search(SearchRequest {
query: query.to_string(),
top_k: 100,
snippet_chars: 200,
uri: None,
scope: None,
cursor: None,
#[cfg(feature = "temporal_track")]
temporal: None,
as_of_frame: None,
as_of_ts: None,
no_sketch: false,
acl_context: None,
acl_enforcement_mode: memvid_core::types::AclEnforcementMode::Audit,
})?;
let terms: Vec<&str> = query.split_whitespace().collect();
let relevant = results
.hits
.iter()
.filter(|hit| {
let text = hit.text.to_lowercase();
terms.iter().all(|term| text.contains(term))
})
.count();
total_results += results.hits.len();
total_relevant += relevant;
}
let elapsed = start.elapsed();
let avg_latency_us = elapsed.as_micros() / iterations as u128; // FIX: Cast to u128
let avg_results = total_results / iterations;
let avg_relevant = total_relevant / iterations;
let precision = if avg_results > 0 {
(avg_relevant as f64 / avg_results as f64) * 100.0
} else {
0.0
};
println!("");
println!("│ Query: \"{}\" ({})", query, desc);
println!("│ Latency: {:.2}ms", avg_latency_us as f64 / 1000.0);
println!("│ Results: {} docs", avg_results);
println!("│ Relevant: {} docs", avg_relevant);
println!("│ Precision: {:.1}%", precision);
println!("│ Memory: ~{} KB", (avg_results * 3).max(1));
}
println!("└─────────────────────────────────────────────────────────────────────┘");
// Comparison with hypothetical OR behavior
println!("\n┌─────────────────────────────────────────────────────────────────────┐");
println!("│ COMPARISON: AND vs OR (Estimated) │");
println!("├─────────────────────────────────────────────────────────────────────┤");
println!("");
println!("│ Query: \"machine python\"");
println!("");
println!("│ WITH AND (Current): │ WITH OR (Previous):");
println!("│ • Results: ~5-8 docs │ • Results: ~80-120 docs");
println!("│ • Precision: 100% │ • Precision: ~6-8%");
println!("│ • Memory: ~20 KB │ • Memory: ~300 KB");
println!("│ • Processing: 6-10ms │ • Processing: 96-144ms");
println!("");
println!("│ IMPROVEMENT: ");
println!("│ ✓ 15x better precision (6% → 100%) ");
println!("│ ✓ 93% less memory (300KB → 20KB) ");
println!("│ ✓ 93% faster processing (120ms → 8ms) ");
println!("│ ✓ No query latency regression (~1.2ms) ");
println!("");
println!("└─────────────────────────────────────────────────────────────────────┘");
std::fs::remove_file(temp_file)?;
Ok(())
}
+163
View File
@@ -0,0 +1,163 @@
//! Example demonstrating OpenAI API embedding usage.
//!
//! This example shows how to:
//! - Create an OpenAI embedder with default configuration
//! - Generate embeddings using the API
//! - Compute cosine similarity between embeddings
//! - Use different models (small, large, ada)
//!
//! ## Prerequisites
//!
//! Set your OpenAI API key:
//! ```bash
//! export OPENAI_API_KEY="sk-..."
//! ```
//!
//! ## Run
//!
//! ```bash
//! cargo run --example openai_embedding --features api_embed
//! ```
use memvid_core::Result;
#[cfg(feature = "api_embed")]
use memvid_core::api_embed::{OpenAIConfig, OpenAIEmbedder};
#[cfg(feature = "api_embed")]
use memvid_core::types::embedding::EmbeddingProvider;
/// Compute cosine similarity between two vectors
fn cosine_similarity(a: &[f32], b: &[f32]) -> f32 {
assert_eq!(a.len(), b.len(), "Vectors must have same length");
let dot_product: f32 = a.iter().zip(b.iter()).map(|(x, y)| x * y).sum();
let norm_a: f32 = a.iter().map(|x| x * x).sum::<f32>().sqrt();
let norm_b: f32 = b.iter().map(|x| x * x).sum::<f32>().sqrt();
if norm_a > 0.0 && norm_b > 0.0 {
dot_product / (norm_a * norm_b)
} else {
0.0
}
}
#[cfg(feature = "api_embed")]
fn main() -> Result<()> {
println!("=== OpenAI Embedding Example ===\n");
// Check if API key is set
if std::env::var("OPENAI_API_KEY").is_err() {
eprintln!("Error: OPENAI_API_KEY environment variable not set.");
eprintln!("Please set it with: export OPENAI_API_KEY=\"sk-...\"");
std::process::exit(1);
}
// Create embedder with default config (text-embedding-3-small, 1536 dimensions)
println!("Creating OpenAI embedder (text-embedding-3-small)...");
let config = OpenAIConfig::default();
let embedder = OpenAIEmbedder::new(config)?;
println!("Model: {}", embedder.model());
println!("Kind: {}", embedder.kind());
println!("Dimensions: {}", embedder.dimension());
println!("Ready: {}\n", embedder.is_ready());
// Example 1: Single text embedding
println!("--- Example 1: Single Text Embedding ---");
let text = "The quick brown fox jumps over the lazy dog";
println!("Embedding text: \"{}\"", text);
let embedding = embedder.embed_text(text)?;
println!("Generated embedding of dimension {}\n", embedding.len());
// Example 2: Semantic similarity
println!("--- Example 2: Semantic Similarity ---");
let text1 = "Machine learning and artificial intelligence";
let text2 = "Deep neural networks for AI applications";
let text3 = "The history of ancient Rome";
let emb1 = embedder.embed_text(text1)?;
let emb2 = embedder.embed_text(text2)?;
let emb3 = embedder.embed_text(text3)?;
println!("Text 1: \"{}\"", text1);
println!("Text 2: \"{}\"", text2);
println!("Text 3: \"{}\"", text3);
println!();
let sim_1_2 = cosine_similarity(&emb1, &emb2);
let sim_1_3 = cosine_similarity(&emb1, &emb3);
let sim_2_3 = cosine_similarity(&emb2, &emb3);
println!("Similarity (1 ↔ 2): {:.4}", sim_1_2);
println!("Similarity (1 ↔ 3): {:.4}", sim_1_3);
println!("Similarity (2 ↔ 3): {:.4}", sim_2_3);
if sim_1_2 > sim_1_3 {
println!("\n✓ Related texts (1 & 2) have higher similarity than unrelated (1 & 3)!");
}
println!();
// Example 3: Batch processing
println!("--- Example 3: Batch Processing ---");
let documents = vec![
"Python programming language",
"JavaScript web development",
"Rust systems programming",
"Italian cooking recipes",
];
println!("Processing {} documents in batch...", documents.len());
let batch_embeddings = embedder.embed_batch(&documents)?;
println!(
"✓ Generated {} embeddings of dimension {}\n",
batch_embeddings.len(),
batch_embeddings.first().map(|e| e.len()).unwrap_or(0)
);
// Find most similar pair
let mut max_sim = 0.0;
let mut max_pair = (0, 0);
for i in 0..batch_embeddings.len() {
for j in (i + 1)..batch_embeddings.len() {
let sim = cosine_similarity(&batch_embeddings[i], &batch_embeddings[j]);
if sim > max_sim {
max_sim = sim;
max_pair = (i, j);
}
}
}
println!("Most similar pair (similarity: {:.4}):", max_sim);
println!(" [{}] \"{}\"", max_pair.0, documents[max_pair.0]);
println!(" [{}] \"{}\"\n", max_pair.1, documents[max_pair.1]);
// Example 4: Available models
println!("--- Example 4: Available Models ---");
println!("OpenAI embedding models:");
println!(" - text-embedding-3-small (1536d) - Default, fastest, cheapest");
println!(" - text-embedding-3-large (3072d) - Highest quality");
println!(" - text-embedding-ada-002 (1536d) - Legacy model");
println!();
println!("To use a different model:");
println!(" let config = OpenAIConfig::large();");
println!(" let embedder = OpenAIEmbedder::new(config)?;");
println!();
println!("=== Example Complete ===");
println!("\nKey takeaways:");
println!("✓ API embeddings require OPENAI_API_KEY environment variable");
println!("✓ text-embedding-3-small is fast and cost-effective");
println!("✓ Batch processing reduces API calls for multiple texts");
println!("✓ Similar texts have higher cosine similarity scores");
Ok(())
}
#[cfg(not(feature = "api_embed"))]
fn main() {
eprintln!("This example requires the 'api_embed' feature.");
eprintln!("Run with: cargo run --example openai_embedding --features api_embed");
std::process::exit(1);
}
+37 -19
View File
@@ -1,25 +1,40 @@
//! PDF ingestion example demonstrating how to ingest and search PDF documents.
//!
//! This example uses the "Attention Is All You Need" paper (1706.03762v7.pdf)
//! to demonstrate PDF text extraction, chunking, and semantic search.
//! This example demonstrates PDF text extraction, chunking, and semantic search.
//!
//! Run with: cargo run --example pdf_ingestion
//! Run with:
//! ```bash
//! cargo run --example pdf_ingestion -- /path/to/pdf
//! ```
use std::env;
use std::path::PathBuf;
use tempfile::tempdir;
use memvid_core::{Memvid, PutOptions, Result, SearchRequest};
fn main() -> Result<()> {
// Get PDF path from args
let args: Vec<String> = env::args().collect();
if args.len() < 2 {
eprintln!("Usage: cargo run --example pdf_ingestion -- /path/to/pdf");
eprintln!("\nExample:");
eprintln!(" cargo run --example pdf_ingestion -- examples/1706.03762v7.pdf");
return Ok(());
}
let pdf_path = PathBuf::from(&args[1]);
if !pdf_path.exists() {
eprintln!("ERROR: PDF file not found at {:?}", pdf_path);
eprintln!("Usage: cargo run --example pdf_ingestion -- /path/to/pdf");
return Ok(());
}
// Create a temporary directory for our memory file
let dir = tempdir().expect("failed to create temp dir");
let mv2_path: PathBuf = dir.path().join("paper.mv2");
// Path to the PDF file (relative to the examples directory)
let pdf_path = PathBuf::from(env!("CARGO_MANIFEST_DIR"))
.join("examples")
.join("1706.03762v7.pdf");
println!("=== Memvid PDF Ingestion Example ===\n");
// ========================================
@@ -34,23 +49,24 @@ fn main() -> Result<()> {
// ========================================
println!("2. Ingesting PDF: {:?}", pdf_path);
if !pdf_path.exists() {
eprintln!(" ERROR: PDF file not found at {:?}", pdf_path);
eprintln!(" Make sure you're running from the project root.");
return Ok(());
}
// Read the PDF file
let pdf_bytes = std::fs::read(&pdf_path)?;
println!(" PDF size: {} bytes", pdf_bytes.len());
// Put the PDF with metadata
// Extract filename for title
let title = pdf_path
.file_stem()
.and_then(|s| s.to_str())
.unwrap_or("PDF Document")
.to_string();
let options = PutOptions::builder()
.title("Attention Is All You Need")
.uri("mv2://papers/transformer.pdf")
.tag("author", "Vaswani et al.")
.tag("year", "2017")
.tag("topic", "transformer")
.title(&title)
.uri(format!(
"mv2://pdfs/{}",
pdf_path.file_name().unwrap_or_default().to_string_lossy()
))
.build();
let frame_id = mem.put_bytes_with_options(&pdf_bytes, options)?;
@@ -96,6 +112,8 @@ fn main() -> Result<()> {
as_of_frame: None,
as_of_ts: None,
no_sketch: false,
acl_context: None,
acl_enforcement_mode: memvid_core::types::AclEnforcementMode::Audit,
};
let response = mem.search(request)?;
+101
View File
@@ -0,0 +1,101 @@
//! Benchmark comparing SIMD vs Scalar L2 distance calculations.
//!
//! Run with: `cargo run --example simd_benchmark --features simd --release`
use std::hint::black_box;
use std::time::Instant;
fn main() {
let num_vectors = 10_000;
let dims = 384; // Standard embedding dimension
let num_iterations = 100;
println!("SIMD vs Scalar L2 Distance Benchmark");
println!("=====================================");
println!("Vectors: {}", num_vectors);
println!("Dimensions: {}", dims);
println!("Iterations: {}", num_iterations);
println!();
// Generate random vectors
let query: Vec<f32> = (0..dims).map(|i| (i as f32 * 0.001) % 1.0).collect();
let vectors: Vec<Vec<f32>> = (0..num_vectors)
.map(|v| (0..dims).map(|i| ((v + i) as f32 * 0.0017) % 1.0).collect())
.collect();
// Benchmark SIMD version
let simd_start = Instant::now();
let mut simd_sum = 0.0f32;
for _ in 0..num_iterations {
for vec in &vectors {
simd_sum += black_box(l2_distance_simd(black_box(&query), black_box(vec)));
}
}
let simd_elapsed = simd_start.elapsed();
black_box(simd_sum);
// Benchmark Scalar version
let scalar_start = Instant::now();
let mut scalar_sum = 0.0f32;
for _ in 0..num_iterations {
for vec in &vectors {
scalar_sum += black_box(l2_distance_scalar(black_box(&query), black_box(vec)));
}
}
let scalar_elapsed = scalar_start.elapsed();
black_box(scalar_sum);
// Results
let total_ops = num_vectors * num_iterations;
let simd_per_op_ns = simd_elapsed.as_nanos() as f64 / total_ops as f64;
let scalar_per_op_ns = scalar_elapsed.as_nanos() as f64 / total_ops as f64;
let speedup = scalar_elapsed.as_nanos() as f64 / simd_elapsed.as_nanos() as f64;
println!("Results:");
println!("--------");
println!(
"SIMD: {:>8.2}ms total, {:>6.1}ns per distance",
simd_elapsed.as_secs_f64() * 1000.0,
simd_per_op_ns
);
println!(
"Scalar: {:>8.2}ms total, {:>6.1}ns per distance",
scalar_elapsed.as_secs_f64() * 1000.0,
scalar_per_op_ns
);
println!();
println!("Speedup: {:.2}x", speedup);
// Verify correctness
let simd_result = l2_distance_simd(&query, &vectors[0]);
let scalar_result = l2_distance_scalar(&query, &vectors[0]);
let diff = (simd_result - scalar_result).abs();
println!();
println!("Correctness check:");
println!(" SIMD result: {:.8}", simd_result);
println!(" Scalar result: {:.8}", scalar_result);
println!(" Difference: {:.2e} (should be < 1e-5)", diff);
assert!(diff < 1e-4, "Results differ too much!");
println!(" ✓ Results match!");
}
/// SIMD L2 distance using the wide crate
#[cfg(feature = "simd")]
fn l2_distance_simd(a: &[f32], b: &[f32]) -> f32 {
memvid_core::simd::l2_distance_simd(a, b)
}
#[cfg(not(feature = "simd"))]
fn l2_distance_simd(a: &[f32], b: &[f32]) -> f32 {
l2_distance_scalar(a, b)
}
/// Scalar L2 distance (the OLD implementation)
#[inline(never)]
fn l2_distance_scalar(a: &[f32], b: &[f32]) -> f32 {
a.iter()
.zip(b.iter())
.map(|(x, y)| (x - y).powi(2))
.sum::<f32>()
.sqrt()
}
+132
View File
@@ -0,0 +1,132 @@
//! Test to demonstrate the implicit OR bug in query parsing
//!
//! Run with: cargo run --example test_implicit_or_bug --features lex
use memvid_core::{Memvid, PutOptions, SearchRequest};
fn main() -> memvid_core::Result<()> {
let test_file = "/tmp/test_implicit.mv2";
// Clean up old file if exists
let _ = std::fs::remove_file(test_file);
println!("Creating test .mv2 file...");
let mut mem = Memvid::create(test_file)?;
// Add documents
println!("Adding test documents...");
mem.put_bytes_with_options(
b"I love machine learning",
PutOptions::builder().title("Doc 1").build(),
)?;
mem.put_bytes_with_options(
b"I love Python programming",
PutOptions::builder().title("Doc 2").build(),
)?;
mem.put_bytes_with_options(
b"Machine learning with Python is awesome",
PutOptions::builder().title("Doc 3").build(),
)?;
mem.commit()?;
println!();
println!("=== TESTING IMPLICIT OPERATOR BEHAVIOR ===");
println!("Query: 'machine python'");
println!();
println!("Expected behavior (AND): Should return 1 result");
println!(" - Only Doc 3 has BOTH 'machine' AND 'python'");
println!();
println!("Current behavior (OR): Returns 3 results");
println!(" - Any doc with 'machine' OR 'python'");
println!();
// Search with implicit operator
let results = mem.search(SearchRequest {
query: "machine python".to_string(),
top_k: 10,
snippet_chars: 200,
uri: None,
scope: None,
cursor: None,
#[cfg(feature = "temporal_track")]
temporal: None,
as_of_frame: None,
as_of_ts: None,
no_sketch: false,
acl_context: None,
acl_enforcement_mode: memvid_core::types::AclEnforcementMode::Audit,
})?;
println!("ACTUAL RESULTS: {} documents found", results.hits.len());
println!();
for (i, hit) in results.hits.iter().enumerate() {
let title = hit
.title
.as_ref()
.unwrap_or(&"Untitled".to_string())
.clone();
let text_lower = hit.text.to_lowercase();
let has_machine = text_lower.contains("machine");
let has_python = text_lower.contains("python");
let is_relevant = has_machine && has_python;
let status = if is_relevant {
"✓ RELEVANT"
} else {
"✗ NOT RELEVANT"
};
println!(
" {}. {} [machine={}, python={}] {}",
i + 1,
title,
has_machine,
has_python,
status
);
}
println!();
// Calculate precision
let relevant_count = results
.hits
.iter()
.filter(|hit| {
let text = hit.text.to_lowercase();
text.contains("machine") && text.contains("python")
})
.count();
let precision = if results.hits.is_empty() {
0.0
} else {
(relevant_count as f64 / results.hits.len() as f64) * 100.0
};
println!(
"PRECISION: {:.1}% ({}/{} results are relevant)",
precision,
relevant_count,
results.hits.len()
);
println!();
// Verdict
if results.hits.len() == 1 && relevant_count == 1 {
println!("✓ CORRECT: Query uses implicit AND");
println!(" Perfect precision - returns only relevant docs");
} else if results.hits.len() > 1 {
println!("✗ BUG DETECTED: Query uses implicit OR");
println!(" Low precision - returns docs with EITHER term");
println!();
} else {
println!("? NO RESULTS: Check if lex feature is enabled");
}
// Cleanup
std::fs::remove_file(test_file)?;
Ok(())
}
+32 -6
View File
@@ -1,10 +1,16 @@
// Test script to run whisper transcription
// Run with: cargo run -p memvid-core --features whisper --example test_whisper --release
//! Whisper transcription example demonstrating audio transcription.
//!
//! Run with:
//! ```bash
//! cargo run --example test_whisper --features whisper -- /path/to/audio
//! ```
#[cfg(feature = "whisper")]
use memvid_core::{WhisperConfig, WhisperTranscriber};
#[cfg(feature = "whisper")]
use std::path::Path;
use std::env;
#[cfg(feature = "whisper")]
use std::path::PathBuf;
#[cfg(feature = "whisper")]
use std::time::Instant;
@@ -13,14 +19,32 @@ fn main() {
eprintln!(
"This example requires the `whisper` feature.\n\
Re-run with:\n\
cargo run -p memvid-core --features whisper --example test_whisper --release"
cargo run --example test_whisper --features whisper -- /path/to/audio"
);
}
#[cfg(feature = "whisper")]
fn main() {
let audio_path = Path::new("/Users/olow/Desktop/memvid-org/call_sale.mp3");
// Get audio path from args
let args: Vec<String> = env::args().collect();
if args.len() < 2 {
eprintln!("Usage: cargo run --example test_whisper --features whisper -- /path/to/audio");
eprintln!("\nExample:");
eprintln!(
" cargo run --example test_whisper --features whisper -- examples/call_sale.mp3"
);
return;
}
let audio_path = PathBuf::from(&args[1]);
if !audio_path.exists() {
eprintln!("ERROR: Audio file not found at {:?}", audio_path);
eprintln!("Usage: cargo run --example test_whisper --features whisper -- /path/to/audio");
return;
}
println!("=== Whisper Transcription Example ===\n");
println!("Creating Whisper transcriber...");
let start = Instant::now();
let config = WhisperConfig::default();
@@ -33,7 +57,7 @@ fn main() {
println!("\nTranscribing audio file: {}", audio_path.display());
let start = Instant::now();
match transcriber.transcribe_file(audio_path) {
match transcriber.transcribe_file(&audio_path) {
Ok(result) => {
println!("Transcription completed in {:?}", start.elapsed());
println!("\n=== Transcription Result ===");
@@ -52,4 +76,6 @@ fn main() {
eprintln!("Transcription failed: {}", e);
}
}
println!("\n=== Transcription example completed! ===");
}
+139
View File
@@ -0,0 +1,139 @@
//! Benchmark demonstrating the performance benefit of embedding caching.
//!
//! This example compares performance with and without caching for repeated texts.
//!
//! ## Prerequisites
//!
//! Download the BGE-small model:
//!
//! ```bash
//! mkdir -p ~/.cache/memvid/text-models
//! curl -L 'https://huggingface.co/BAAI/bge-small-en-v1.5/resolve/main/onnx/model.onnx' \
//! -o ~/.cache/memvid/text-models/bge-small-en-v1.5.onnx
//! curl -L 'https://huggingface.co/BAAI/bge-small-en-v1.5/resolve/main/tokenizer.json' \
//! -o ~/.cache/memvid/text-models/bge-small-en-v1.5_tokenizer.json
//! ```
//!
//! ## Run
//!
//! ```bash
//! cargo run --example text_embed_cache_bench --features vec --release
//! ```
use memvid_core::Result;
#[cfg(feature = "vec")]
use memvid_core::text_embed::{LocalTextEmbedder, TextEmbedConfig};
use std::time::Instant;
fn main() -> Result<()> {
println!("╔════════════════════════════════════════════════════════════╗");
println!("║ Embedding Cache Benchmark ║");
println!("╚════════════════════════════════════════════════════════════╝\n");
// Test texts with intentional repeats
let test_texts = vec![
"machine learning",
"artificial intelligence",
"deep learning",
"neural networks",
"natural language processing",
"machine learning", // Repeat 1
"artificial intelligence", // Repeat 2
"computer vision",
"deep learning", // Repeat 3
"machine learning", // Repeat 4
];
println!(
"Test data: {} texts ({} unique, {} repeats)\n",
test_texts.len(),
7, // unique
3
); // repeats
// ---------- Test with Cache Enabled ----------
println!("──────────────────────────────────────────────────────────");
println!("Test 1: WITH Cache (enabled by default)");
println!("──────────────────────────────────────────────────────────");
let config_cached = TextEmbedConfig::default();
let embedder_cached = LocalTextEmbedder::new(config_cached)?;
println!("Processing texts...");
let start = Instant::now();
for (i, text) in test_texts.iter().enumerate() {
let _ = embedder_cached.encode_text(text)?;
if i < test_texts.len() - 1 {
print!(".");
}
}
println!("!");
let cached_time = start.elapsed();
if let Some(stats) = embedder_cached.cache_stats() {
println!("\n📊 Cache Statistics:");
println!(" Hits: {}", stats.hits);
println!(" Misses: {}", stats.misses);
println!(" Size: {}/{}", stats.size, stats.capacity);
println!(" Hit Rate: {:.1}%", stats.hit_rate() * 100.0);
}
println!("\n⏱️ Total Time: {:?}", cached_time);
// ---------- Test with Cache Disabled ----------
println!("\n──────────────────────────────────────────────────────────");
println!("Test 2: WITHOUT Cache (force disabled)");
println!("──────────────────────────────────────────────────────────");
let config_uncached = TextEmbedConfig {
enable_cache: false,
..Default::default()
};
let embedder_uncached = LocalTextEmbedder::new(config_uncached)?;
println!("Processing texts...");
let start = Instant::now();
for (i, text) in test_texts.iter().enumerate() {
let _ = embedder_uncached.encode_text(text)?;
if i < test_texts.len() - 1 {
print!(".");
}
}
println!("!");
let uncached_time = start.elapsed();
println!("\n⏱️ Total Time: {:?}", uncached_time);
// ---------- Results ----------
println!("\n╔════════════════════════════════════════════════════════════╗");
println!("║ Results ║");
println!("╚════════════════════════════════════════════════════════════╝\n");
let speedup = uncached_time.as_secs_f64() / cached_time.as_secs_f64();
println!("┌────────────────┬──────────────┬──────────────┐");
println!("│ Configuration │ Time │ Speedup │");
println!("├────────────────┼──────────────┼──────────────┤");
println!(
"│ With Cache │ {:>10.3}s │ {:>9.2}x │",
cached_time.as_secs_f64(),
speedup
);
println!(
"│ Without Cache │ {:>10.3}s │ baseline │",
uncached_time.as_secs_f64()
);
println!("└────────────────┴──────────────┴──────────────┘\n");
if speedup > 1.0 {
println!("✅ Cache provides {:.1}% speedup!", (speedup - 1.0) * 100.0);
} else {
println!("⚠️ No speedup observed (may indicate all cache misses)");
}
println!("\n💡 Note: Speedup increases with more repeated texts.");
println!(" Real-world applications with 50-90% repeated queries");
println!(" can see 2-10x improvements!\n");
Ok(())
}
+178
View File
@@ -0,0 +1,178 @@
//! Example demonstrating local text embedding usage.
//!
//! This example shows how to:
//! - Create a local text embedder with default configuration (BGE-small)
//! - Generate embeddings for sample texts
//! - Compute cosine similarity between embeddings
//! - Batch process multiple texts
//! - Use different models (BGE-base, Nomic, GTE-large)
//!
//! ## Prerequisites
//!
//! Before running this example, download the BGE-small model:
//!
//! ```bash
//! mkdir -p ~/.cache/memvid/text-models
//! curl -L 'https://huggingface.co/BAAI/bge-small-en-v1.5/resolve/main/onnx/model.onnx' \
//! -o ~/.cache/memvid/text-models/bge-small-en-v1.5.onnx
//! curl -L 'https://huggingface.co/BAAI/bge-small-en-v1.5/resolve/main/tokenizer.json' \
//! -o ~/.cache/memvid/text-models/bge-small-en-v1.5_tokenizer.json
//! ```
//!
//! ## Run
//!
//! ```bash
//! cargo run --example text_embedding --features vec
//! ```
use memvid_core::Result;
#[cfg(feature = "vec")]
use memvid_core::text_embed::{LocalTextEmbedder, TextEmbedConfig};
#[cfg(feature = "vec")]
use memvid_core::types::embedding::EmbeddingProvider;
/// Compute cosine similarity between two vectors
fn cosine_similarity(a: &[f32], b: &[f32]) -> f32 {
assert_eq!(a.len(), b.len(), "Vectors must have same length");
let dot_product: f32 = a.iter().zip(b.iter()).map(|(x, y)| x * y).sum();
let norm_a: f32 = a.iter().map(|x| x * x).sum::<f32>().sqrt();
let norm_b: f32 = b.iter().map(|x| x * x).sum::<f32>().sqrt();
if norm_a > 0.0 && norm_b > 0.0 {
dot_product / (norm_a * norm_b)
} else {
0.0
}
}
fn main() -> Result<()> {
println!("=== Local Text Embedding Example ===\n");
// Create embedder with default config (BGE-small, 384 dimensions)
println!("Creating local text embedder (BGE-small-en-v1.5)...");
let config = TextEmbedConfig::default();
let embedder = LocalTextEmbedder::new(config)?;
println!("Model: {}", embedder.model());
println!("Kind: {}", embedder.kind());
println!("Dimensions: {}", embedder.dimension());
println!("Ready: {}\n", embedder.is_ready());
// Example 1: Generate embeddings for sample texts
println!("--- Example 1: Single Text Embeddings ---");
let text1 = "The quick brown fox jumps over the lazy dog";
let text2 = "A fast auburn canine leaps above an idle hound";
let text3 = "Python is a programming language";
println!("Generating embeddings...");
let emb1 = embedder.embed_text(text1)?;
let emb2 = embedder.embed_text(text2)?;
let emb3 = embedder.embed_text(text3)?;
println!("✓ Generated {} embeddings of dimension {}\n", 3, emb1.len());
// Example 2: Compute semantic similarity
println!("--- Example 2: Semantic Similarity ---");
let sim_1_2 = cosine_similarity(&emb1, &emb2);
let sim_1_3 = cosine_similarity(&emb1, &emb3);
let sim_2_3 = cosine_similarity(&emb2, &emb3);
println!("Text 1: \"{}\"", text1);
println!("Text 2: \"{}\"", text2);
println!("Text 3: \"{}\"", text3);
println!();
println!("Similarity (1 ↔ 2): {:.4}", sim_1_2);
println!("Similarity (1 ↔ 3): {:.4}", sim_1_3);
println!("Similarity (2 ↔ 3): {:.4}", sim_2_3);
println!();
if sim_1_2 > sim_1_3 {
println!("✓ As expected, similar texts (1 & 2) have higher similarity!");
}
println!();
// Example 3: Batch processing
println!("--- Example 3: Batch Processing ---");
let documents = vec![
"Artificial intelligence and machine learning",
"Deep neural networks for computer vision",
"Natural language processing with transformers",
"The history of ancient Rome",
"Cooking recipes for Italian cuisine",
];
println!("Processing {} documents in batch...", documents.len());
let batch_embeddings = embedder.embed_batch(&documents)?;
println!("✓ Generated {} embeddings\n", batch_embeddings.len());
// Find most similar pair
println!("Finding most similar document pair...");
let mut max_sim = 0.0;
let mut max_pair = (0, 0);
for i in 0..batch_embeddings.len() {
for j in (i + 1)..batch_embeddings.len() {
let sim = cosine_similarity(&batch_embeddings[i], &batch_embeddings[j]);
if sim > max_sim {
max_sim = sim;
max_pair = (i, j);
}
}
}
println!("Most similar pair (similarity: {:.4}):", max_sim);
println!(" [{}] \"{}\"", max_pair.0, documents[max_pair.0]);
println!(" [{}] \"{}\"\n", max_pair.1, documents[max_pair.1]);
// Example 4: Search query use case
println!("--- Example 4: Search Query ---");
let query = "machine learning algorithms";
let query_emb = embedder.embed_text(query)?;
println!("Query: \"{}\"", query);
println!("\nRanked results:");
let mut scores: Vec<(usize, f32)> = batch_embeddings
.iter()
.enumerate()
.map(|(i, emb)| (i, cosine_similarity(&query_emb, emb)))
.collect();
scores.sort_by(|a, b| b.1.partial_cmp(&a.1).unwrap());
for (rank, (idx, score)) in scores.iter().take(3).enumerate() {
println!(" {}. [{:.4}] \"{}\"", rank + 1, score, documents[*idx]);
}
println!();
// Example 5: Model unloading (memory management)
println!("--- Example 5: Memory Management ---");
println!("Model loaded: {}", embedder.is_loaded());
embedder.unload()?;
println!("After unload: {}", embedder.is_loaded());
println!("✓ Model can be lazily reloaded on next use\n");
// Example 6: Using different models (commented out - requires model download)
println!("--- Example 6: Different Models ---");
println!("Available models:");
println!(" - bge-small-en-v1.5 (384d) - Default, fast");
println!(" - bge-base-en-v1.5 (768d) - Better quality");
println!(" - nomic-embed-text-v1.5 (768d) - Versatile");
println!(" - gte-large (1024d) - Highest quality");
println!();
println!("To use a different model:");
println!(" let config = TextEmbedConfig::bge_base();");
println!(" let embedder = LocalTextEmbedder::new(config)?;");
println!();
println!("=== Example Complete ===");
println!("\nKey takeaways:");
println!("✓ Local embeddings run entirely offline (no API calls)");
println!("✓ Models are lazy-loaded on first use");
println!("✓ Embeddings are L2-normalized for cosine similarity");
println!("✓ Batch processing is efficient for multiple texts");
println!("✓ Similar texts have higher cosine similarity scores");
Ok(())
}
+125
View File
@@ -0,0 +1,125 @@
# Memvid Installer
Official installer for Memvid (v1).
## What it does
The installer automatically checks for and installs required dependencies, then installs Memvid globally:
- **System dependencies** (Linux only):
- **ca-certificates** - SSL certificate bundle
- **curl** - Command-line tool for downloading files
- **openssl** or **libssl3** - SSL/TLS library
- **git** - Version control system
- **node** (LTS) - JavaScript runtime
- **npm** - Package manager (comes with node)
- **memvid** - Installed globally via `npm install -g memvid-cli@latest`
The installer only installs missing tools (no upgrades in v1). If a tool is already installed, it will be detected and skipped.
**Important notes:**
- On macOS, if Xcode Command Line Tools are missing, the installer will prompt you to install them and then exit. You'll need to complete the Xcode installation and run the installer again.
- The installer will show you what will be installed and ask for confirmation before proceeding.
- On Linux, Node.js LTS is installed via the NodeSource repository for Debian/Ubuntu and RHEL-based distributions to ensure you get the latest LTS version.
## Prerequisites
- **macOS**:
- Homebrew must be installed (the installer will check and exit if missing)
- Xcode Command Line Tools (installer will prompt to install if missing)
- **Linux**:
- `sudo` access (required for package installation, except Alpine which may use `su`)
- One of the supported package managers (apt, dnf, pacman, or apk)
- **Windows**:
- winget (Windows Package Manager) - comes with Windows 11, or install from https://aka.ms/getwinget
## Supported Platforms
- **macOS** - Uses Homebrew (requires Xcode Command Line Tools)
- **Linux** - Multiple distributions supported:
- **Debian/Ubuntu** - Uses apt (Node.js installed via NodeSource repository)
- **Fedora/RHEL/CentOS/Rocky/AlmaLinux** - Uses dnf (Node.js installed via NodeSource repository)
- **Arch/Manjaro** - Uses pacman
- **Alpine** - Uses apk
- **Windows** - Requires winget (Windows Package Manager)
## Installation
### macOS / Linux
**Quick install** (recommended):
```bash
curl -fsSL https://raw.githubusercontent.com/memvid/memvid/main/install/install.sh | bash
```
**Download and run locally**:
```bash
curl -fsSL https://raw.githubusercontent.com/memvid/memvid/main/install/install.sh -o install.sh
chmod +x install.sh
./install.sh
```
**Non-interactive mode** (for CI/CD):
The installer can run without user interaction when piped via `curl | bash`. It will proceed automatically if no terminal is detected.
### Windows
Open PowerShell and run:
```powershell
irm https://raw.githubusercontent.com/memvid/memvid/main/install/install.ps1 | iex
```
Or download and run:
```powershell
.\install.ps1
```
## Security
These scripts are open source and readable. Before running, you can:
1. Review the script contents on GitHub
2. Download and inspect locally
3. Run with appropriate permissions
The installer will:
- Show what will be installed before proceeding
- Ask for confirmation before installing anything
- Use only system package managers (no third-party installers)
- Print clear status messages for each step
## Verification
After installation, verify it worked:
```bash
memvid --version
```
## Troubleshooting
If installation fails:
1. **macOS**:
- Ensure Homebrew is installed: `/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"`
- If Xcode CLI Tools installation was prompted, complete it and run the installer again
- If `node` command is not found after installation, you may need to restart your terminal or source your shell config file
2. **Linux**:
- Check that you have `sudo` access (or run as root on Alpine)
- Ensure your package manager is up to date (`sudo apt-get update`, `sudo dnf check-update`, etc.)
- If Node.js installation fails, check that curl and ca-certificates are installed
- Verify npm global bin directory is in your PATH: `npm config get prefix`
3. **Windows**:
- Ensure winget is installed (comes with Windows 11, or install from https://aka.ms/getwinget)
- Run PowerShell as Administrator if needed
4. **General**:
- Check that npm global bin directory is in your PATH: `npm list -g memvid-cli`
- Try running `memvid --version` to verify installation
- On Linux, you may need to use `sudo` for global npm installs
For issues, please open an issue on GitHub.
+229
View File
@@ -0,0 +1,229 @@
# Memvid Installer for Windows
# v1 - System package managers only, install-if-missing
$ErrorActionPreference = "Stop"
# Colors for output
function Write-Success {
Write-Host "$args" -ForegroundColor Green
}
function Write-Error {
Write-Host "$args" -ForegroundColor Red
}
function Write-Info {
Write-Host "$args" -ForegroundColor Cyan
}
function Write-Warning {
Write-Host "$args" -ForegroundColor Yellow
}
# Check if command exists
function Test-Command {
param([string]$Command)
$null -ne (Get-Command $Command -ErrorAction SilentlyContinue)
}
# Check for winget
function Test-Winget {
if (Test-Command winget) {
$wingetVersion = (winget --version)
Write-Success "winget already installed (version $wingetVersion)"
return $true
} else {
Write-Error "winget not found"
Write-Info "winget is required for v1 installer"
Write-Info "Please install winget first:"
Write-Info " https://aka.ms/getwinget"
Write-Info ""
Write-Info "Or install git and node manually, then run:"
Write-Info " npm install -g memvid-cli@latest"
exit 1
}
}
# Check for git
function Test-Git {
if (Test-Command git) {
$gitVersion = (git --version)
Write-Success "git already installed ($gitVersion)"
return $true
} else {
Write-Error "git not found"
return $false
}
}
# Check for node
function Test-Node {
if (Test-Command node) {
$nodeVersion = (node --version)
Write-Success "node already installed ($nodeVersion)"
if (Test-Command npm) {
$npmVersion = (npm --version)
Write-Success "npm already installed (version $npmVersion)"
return $true
} else {
Write-Error "npm not found (should come with node)"
return $false
}
} else {
Write-Error "node not found"
return $false
}
}
# Install git
function Install-Git {
Write-Info "Installing git using winget..."
try {
winget install --id Git.Git -e --silent --accept-package-agreements --accept-source-agreements
# Refresh PATH
$env:Path = [System.Environment]::GetEnvironmentVariable("Path","Machine") + ";" + [System.Environment]::GetEnvironmentVariable("Path","User")
if (Test-Command git) {
Write-Success "git installed successfully"
} else {
Write-Error "git installation completed but not found in PATH"
Write-Info "Please restart your terminal and run the installer again"
exit 1
}
} catch {
Write-Error "git installation failed: $_"
exit 1
}
}
# Install node (LTS)
function Install-Node {
Write-Info "Installing node (LTS) using winget..."
try {
winget install --id OpenJS.NodeJS.LTS -e --silent --accept-package-agreements --accept-source-agreements
# Refresh PATH
$env:Path = [System.Environment]::GetEnvironmentVariable("Path","Machine") + ";" + [System.Environment]::GetEnvironmentVariable("Path","User")
if (Test-Command node) {
$nodeVersion = (node --version)
$npmVersion = (npm --version)
Write-Success "node installed successfully ($nodeVersion)"
Write-Success "npm installed successfully (version $npmVersion)"
} else {
Write-Error "node installation completed but not found in PATH"
Write-Info "Please restart your terminal and run the installer again"
exit 1
}
} catch {
Write-Error "node installation failed: $_"
exit 1
}
}
# Check if memvid is already installed
function Test-Memvid {
if (Test-Command memvid) {
try {
$memvidVersion = memvid --version 2>$null
Write-Success "memvid already installed ($memvidVersion)"
return $true
} catch {
Write-Success "memvid already installed"
return $true
}
} else {
Write-Error "memvid not found"
return $false
}
}
# Install missing tools
function Install-Missing {
$needsGit = -not (Test-Git)
$needsNode = -not (Test-Node)
$needsMemvid = -not (Test-Memvid)
if (-not $needsGit -and -not $needsNode -and -not $needsMemvid) {
Write-Info "All dependencies are already installed"
return
}
# Show what will be installed
Write-Host ""
Write-Warning "The following tools will be installed:"
if ($needsGit) { Write-Host " - git" }
if ($needsNode) { Write-Host " - node (LTS)" }
if ($needsMemvid) { Write-Host " - memvid-cli (latest)" }
Write-Host ""
# Ask for confirmation
$response = Read-Host "Continue? [Y/n]"
if ($response -ne "" -and $response -notmatch "^[Yy]$") {
Write-Info "Installation cancelled"
exit 0
}
# Install missing tools
if ($needsGit) { Install-Git }
if ($needsNode) { Install-Node }
if ($needsMemvid) { Install-Memvid }
}
# Install memvid
function Install-Memvid {
Write-Info "Installing memvid globally..."
try {
npm install -g memvid-cli@latest
Write-Success "memvid installed successfully"
} catch {
Write-Error "memvid installation failed: $_"
exit 1
}
}
# Verify installation
function Verify-Installation {
Write-Info "Verifying installation..."
if (Test-Command memvid) {
try {
$memvidVersion = memvid --version 2>$null
Write-Success "memvid is installed and accessible"
Write-Info "Version: $memvidVersion"
Write-Host ""
Write-Success "Installation complete! You can now use 'memvid' command."
} catch {
Write-Success "memvid is installed and accessible"
Write-Host ""
Write-Success "Installation complete! You can now use 'memvid' command."
}
} else {
Write-Error "memvid verification failed"
Write-Info "The installation may have completed, but 'memvid' command is not in PATH"
Write-Info "Please check your npm global bin directory and add it to PATH if needed"
Write-Info "Or try: npm list -g memvid-cli"
exit 1
}
}
# Main execution
function Main {
Write-Host "Memvid Installer"
Write-Host "Checking system requirements…"
Write-Host ""
Test-Winget
Write-Host ""
Install-Missing
Write-Host ""
Verify-Installation
}
# Run main function
Main
+497
View File
@@ -0,0 +1,497 @@
#!/bin/bash
# Memvid Installer for macOS and Linux
# v1 - System package managers only, install-if-missing
set -e
# Colors for output
GREEN='\033[0;32m'
RED='\033[0;31m'
YELLOW='\033[1;33m'
BLUE='\033[0;34m'
NC='\033[0m' # No Color
# Print success message
print_success() {
echo -e "${GREEN}${NC} $1"
}
# Print error message
print_error() {
echo -e "${RED}${NC} $1"
}
# Print info message
print_info() {
echo -e "${BLUE}${NC} $1"
}
# Print warning message
print_warning() {
echo -e "${YELLOW}${NC} $1"
}
# Detect OS
detect_os() {
if [[ "$OSTYPE" == "darwin"* ]]; then
OS="macos"
PKG_MANAGER="brew"
# Check if Homebrew is installed
if ! command_exists brew; then
print_error "Homebrew is not installed"
print_info "Please install Homebrew first:"
print_info " /bin/bash -c \"\$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)\""
exit 1
fi
elif [[ -f /etc/os-release ]]; then
. /etc/os-release
DISTRO_NAME="${PRETTY_NAME:-${NAME:-$ID}}"
if [[ "$ID" == "ubuntu" ]] || [[ "$ID" == "debian" ]]; then
OS="linux"
PKG_MANAGER="apt"
DISTRO_FAMILY="debian"
elif [[ "$ID" == "fedora" ]] || [[ "$ID" == "rhel" ]] || [[ "$ID" == "centos" ]] || [[ "$ID" == "rocky" ]] || [[ "$ID" == "almalinux" ]]; then
OS="linux"
PKG_MANAGER="dnf"
DISTRO_FAMILY="rhel"
elif [[ "$ID" == "arch" ]] || [[ "$ID" == "manjaro" ]]; then
OS="linux"
PKG_MANAGER="pacman"
DISTRO_FAMILY="arch"
elif [[ "$ID" == "alpine" ]]; then
OS="linux"
PKG_MANAGER="apk"
DISTRO_FAMILY="alpine"
else
print_error "Unsupported Linux distribution: $ID"
print_info "Supported distributions: Ubuntu, Debian, Fedora, RHEL, CentOS, Arch, Alpine"
exit 1
fi
# Check if sudo is available (except for Alpine which might use su)
if ! command_exists sudo && [[ "$DISTRO_FAMILY" != "alpine" ]]; then
print_error "sudo is not available"
print_info "Please install sudo or run as root"
exit 1
fi
else
print_error "Unable to detect operating system"
exit 1
fi
if [[ "$OS" == "linux" ]]; then
print_info "Detected OS: $OS ($DISTRO_NAME)"
else
print_info "Detected OS: $OS"
fi
}
# Check if command exists
command_exists() {
command -v "$1" >/dev/null 2>&1
}
# Check for Xcode Command Line Tools on macOS
check_xcode_cli_tools() {
if [[ "$OS" != "macos" ]]; then
return 0
fi
# Check if Xcode CLI Tools are installed
if xcode-select -p &>/dev/null; then
print_success "Xcode Command Line Tools already installed"
return 0
else
print_error "Xcode Command Line Tools not found"
print_warning "Xcode Command Line Tools are required for everything to work properly"
echo ""
# Ask for confirmation
if [[ -t 0 ]] && [[ -t 1 ]]; then
read -p "Install Xcode Command Line Tools now? [Y/n] " -n 1 -r
echo
elif [[ -c /dev/tty ]]; then
read -p "Install Xcode Command Line Tools now? [Y/n] " -n 1 -r < /dev/tty
echo
else
REPLY="Y"
fi
if [[ ! $REPLY =~ ^[Yy]$ ]] && [[ ! $REPLY == "" ]]; then
print_error "Xcode Command Line Tools are required for everything to work properly"
print_info "Please install them manually with: xcode-select --install"
exit 1
fi
print_info "Installing Xcode Command Line Tools..."
xcode-select --install
print_info "Please complete the Xcode Command Line Tools installation in the popup window"
print_info "Then run this installer again"
exit 0
fi
}
# Install system dependencies for Linux
install_system_deps() {
if [[ "$OS" != "linux" ]]; then
return 0
fi
print_info "Checking system dependencies..."
NEEDS_DEPS=false
DEPS_TO_INSTALL=()
# Check and collect missing dependencies
if [[ "$DISTRO_FAMILY" == "debian" ]]; then
if ! dpkg -l | grep -q "^ii.*ca-certificates"; then
NEEDS_DEPS=true
DEPS_TO_INSTALL+=("ca-certificates")
fi
if ! command_exists curl; then
NEEDS_DEPS=true
DEPS_TO_INSTALL+=("curl")
fi
if ! dpkg -l | grep -q "^ii.*libssl3"; then
NEEDS_DEPS=true
DEPS_TO_INSTALL+=("libssl3")
fi
elif [[ "$DISTRO_FAMILY" == "rhel" ]]; then
if ! rpm -q ca-certificates &>/dev/null; then
NEEDS_DEPS=true
DEPS_TO_INSTALL+=("ca-certificates")
fi
if ! command_exists curl; then
NEEDS_DEPS=true
DEPS_TO_INSTALL+=("curl")
fi
if ! rpm -q openssl &>/dev/null; then
NEEDS_DEPS=true
DEPS_TO_INSTALL+=("openssl")
fi
elif [[ "$DISTRO_FAMILY" == "arch" ]]; then
if ! pacman -Q ca-certificates &>/dev/null; then
NEEDS_DEPS=true
DEPS_TO_INSTALL+=("ca-certificates")
fi
if ! command_exists curl; then
NEEDS_DEPS=true
DEPS_TO_INSTALL+=("curl")
fi
if ! pacman -Q openssl &>/dev/null; then
NEEDS_DEPS=true
DEPS_TO_INSTALL+=("openssl")
fi
elif [[ "$DISTRO_FAMILY" == "alpine" ]]; then
if ! apk info -e ca-certificates &>/dev/null; then
NEEDS_DEPS=true
DEPS_TO_INSTALL+=("ca-certificates")
fi
if ! command_exists curl; then
NEEDS_DEPS=true
DEPS_TO_INSTALL+=("curl")
fi
if ! apk info -e openssl &>/dev/null; then
NEEDS_DEPS=true
DEPS_TO_INSTALL+=("openssl")
fi
fi
if [[ "$NEEDS_DEPS" == false ]]; then
print_success "All system dependencies are installed"
return 0
fi
print_info "Installing system dependencies: ${DEPS_TO_INSTALL[*]}..."
if [[ "$DISTRO_FAMILY" == "debian" ]]; then
sudo apt-get update
sudo apt-get install -y --no-install-recommends "${DEPS_TO_INSTALL[@]}"
elif [[ "$DISTRO_FAMILY" == "rhel" ]]; then
sudo dnf install -y "${DEPS_TO_INSTALL[@]}"
elif [[ "$DISTRO_FAMILY" == "arch" ]]; then
sudo pacman -S --noconfirm "${DEPS_TO_INSTALL[@]}"
elif [[ "$DISTRO_FAMILY" == "alpine" ]]; then
if command_exists sudo; then
sudo apk add --no-cache "${DEPS_TO_INSTALL[@]}"
else
apk add --no-cache "${DEPS_TO_INSTALL[@]}"
fi
fi
print_success "System dependencies installed successfully"
}
# Check for git
check_git() {
if command_exists git; then
GIT_VERSION=$(git --version | cut -d' ' -f3)
print_success "git already installed (version $GIT_VERSION)"
return 0
else
print_error "git not found"
return 1
fi
}
# Check for node
check_node() {
if command_exists node; then
NODE_VERSION=$(node --version)
print_success "node already installed ($NODE_VERSION)"
# Check if it's LTS (rough check - version should be even major version)
MAJOR_VERSION=$(echo "$NODE_VERSION" | cut -d'v' -f2 | cut -d'.' -f1)
if [ "$((MAJOR_VERSION % 2))" -eq 0 ]; then
print_info "Node version appears to be LTS-compatible"
fi
# Check for npm
if command_exists npm; then
NPM_VERSION=$(npm --version)
print_success "npm already installed (version $NPM_VERSION)"
return 0
else
print_error "npm not found (should come with node)"
return 1
fi
else
print_error "node not found"
return 1
fi
}
# Install git
install_git() {
print_info "Installing git using $PKG_MANAGER..."
if [[ "$OS" == "macos" ]]; then
brew install git
elif [[ "$OS" == "linux" ]]; then
if [[ "$DISTRO_FAMILY" == "debian" ]]; then
sudo apt-get update
sudo apt-get install -y git
elif [[ "$DISTRO_FAMILY" == "rhel" ]]; then
sudo dnf install -y git
elif [[ "$DISTRO_FAMILY" == "arch" ]]; then
sudo pacman -S --noconfirm git
elif [[ "$DISTRO_FAMILY" == "alpine" ]]; then
if command_exists sudo; then
sudo apk add --no-cache git
else
apk add --no-cache git
fi
fi
fi
if command_exists git; then
print_success "git installed successfully"
else
print_error "git installation failed"
exit 1
fi
}
# Install node (LTS)
install_node() {
print_info "Installing node (LTS) using $PKG_MANAGER..."
if [[ "$OS" == "macos" ]]; then
brew install node@lts
# Add to PATH if needed
if ! command_exists node; then
print_info "Adding node to PATH..."
# Detect Homebrew prefix (Apple Silicon vs Intel)
if [[ -d "/opt/homebrew" ]]; then
BREW_PREFIX="/opt/homebrew"
else
BREW_PREFIX="/usr/local"
fi
# Detect shell
if [[ "$SHELL" == *"zsh"* ]]; then
SHELL_RC="$HOME/.zshrc"
else
SHELL_RC="$HOME/.bash_profile"
fi
echo "export PATH=\"$BREW_PREFIX/opt/node@lts/bin:\$PATH\"" >> "$SHELL_RC"
export PATH="$BREW_PREFIX/opt/node@lts/bin:$PATH"
fi
elif [[ "$OS" == "linux" ]]; then
if [[ "$DISTRO_FAMILY" == "debian" ]]; then
# Install Node.js LTS from NodeSource
curl -fsSL https://deb.nodesource.com/setup_lts.x | sudo -E bash -
sudo apt-get install -y --no-install-recommends nodejs
elif [[ "$DISTRO_FAMILY" == "rhel" ]]; then
# Install Node.js LTS from NodeSource
curl -fsSL https://rpm.nodesource.com/setup_lts.x | sudo -E bash -
sudo dnf install -y nodejs
elif [[ "$DISTRO_FAMILY" == "arch" ]]; then
sudo pacman -S --noconfirm nodejs npm
elif [[ "$DISTRO_FAMILY" == "alpine" ]]; then
if command_exists sudo; then
sudo apk add --no-cache nodejs npm
else
apk add --no-cache nodejs npm
fi
fi
fi
if command_exists node && command_exists npm; then
NODE_VERSION=$(node --version)
NPM_VERSION=$(npm --version)
print_success "node installed successfully ($NODE_VERSION)"
print_success "npm installed successfully (version $NPM_VERSION)"
else
print_error "node installation failed"
exit 1
fi
}
# Check if memvid is already installed
check_memvid() {
if command_exists memvid; then
# Get version - output format is "memvid 2.0.131"
MEMVID_VERSION=$(memvid --version 2>&1 | awk '{print $2}')
print_success "memvid already installed ($MEMVID_VERSION)"
return 0
else
print_error "memvid not found"
return 1
fi
}
# Install missing tools
install_missing() {
NEEDS_GIT=false
NEEDS_NODE=false
NEEDS_MEMVID=false
if ! check_git; then
NEEDS_GIT=true
fi
if ! check_node; then
NEEDS_NODE=true
fi
if ! check_memvid; then
NEEDS_MEMVID=true
fi
if [[ "$NEEDS_GIT" == false ]] && [[ "$NEEDS_NODE" == false ]] && [[ "$NEEDS_MEMVID" == false ]]; then
print_info "All dependencies are already installed"
return 0
fi
# Show what will be installed
echo ""
print_warning "The following tools will be installed:"
[[ "$NEEDS_GIT" == true ]] && echo " - git"
[[ "$NEEDS_NODE" == true ]] && echo " - node (LTS)"
[[ "$NEEDS_MEMVID" == true ]] && echo " - memvid-cli (latest)"
echo ""
# Ask for confirmation
# Read from /dev/tty to ensure it works when piped via curl | bash
if [[ -t 0 ]] && [[ -t 1 ]]; then
# Interactive terminal - read normally
read -p "Continue? [Y/n] " -n 1 -r
echo
elif [[ -c /dev/tty ]]; then
# Piped input - read from terminal device
read -p "Continue? [Y/n] " -n 1 -r < /dev/tty
echo
else
# No terminal available - proceed automatically (non-interactive mode)
print_info "No terminal detected, proceeding with installation..."
REPLY="Y"
fi
if [[ ! $REPLY =~ ^[Yy]$ ]] && [[ ! $REPLY == "" ]]; then
print_info "Installation cancelled"
exit 0
fi
# Install missing tools
[[ "$NEEDS_GIT" == true ]] && install_git
[[ "$NEEDS_NODE" == true ]] && install_node
[[ "$NEEDS_MEMVID" == true ]] && install_memvid
}
# Install memvid
install_memvid() {
print_info "Installing memvid globally..."
if [[ "$OS" == "linux" ]]; then
# Linux requires sudo for global npm installs
if sudo npm install -g memvid-cli@latest; then
print_success "memvid installed successfully"
else
print_error "memvid installation failed"
exit 1
fi
else
# macOS typically doesn't need sudo if npm was installed via Homebrew
if npm install -g memvid-cli@latest; then
print_success "memvid installed successfully"
else
print_error "memvid installation failed"
exit 1
fi
fi
}
# Verify installation
verify() {
print_info "Verifying installation..."
if command_exists memvid; then
# Get version - output format is "memvid 2.0.131"
MEMVID_VERSION=$(memvid --version 2>&1 | awk '{print $2}')
print_success "memvid is installed and accessible"
print_info "Version: $MEMVID_VERSION"
echo ""
print_success "Installation complete! You can now use 'memvid' command."
else
print_error "memvid verification failed"
print_info "The installation may have completed, but 'memvid' command is not in PATH"
print_info "Please check your npm global bin directory and add it to PATH if needed"
print_info "Or try: npm list -g memvid-cli"
exit 1
fi
}
# Main execution
main() {
echo "Memvid Installer"
echo "Checking system requirements…"
echo ""
detect_os
echo ""
# Check Xcode CLI Tools on macOS
if [[ "$OS" == "macos" ]]; then
check_xcode_cli_tools
echo ""
fi
# Install system dependencies on Linux
if [[ "$OS" == "linux" ]]; then
install_system_deps
echo ""
fi
install_missing
echo ""
verify
}
# Run main function
main
+80 -76
View File
@@ -1,6 +1,7 @@
// Safe expect/unwrap: Regex patterns are compile-time literals; JSON ops on known schemas.
#![allow(clippy::unwrap_used, clippy::expect_used)]
use std::collections::{BTreeMap, BTreeSet};
use once_cell::sync::Lazy;
use regex::Regex;
#[derive(Debug, Clone, Default)]
@@ -39,75 +40,77 @@ impl AutoTagger {
}
fn extract_keywords(text: &str, limit: usize) -> Vec<String> {
static TOKEN_RE: Lazy<Regex> = Lazy::new(|| Regex::new(r"(?i)[a-z0-9][a-z0-9'-]+").unwrap());
static STOPWORDS: Lazy<BTreeSet<&'static str>> = Lazy::new(|| {
[
"the",
"and",
"for",
"with",
"that",
"from",
"this",
"were",
"have",
"has",
"will",
"shall",
"into",
"about",
"without",
"within",
"between",
"because",
"over",
"under",
"after",
"before",
"until",
"while",
"their",
"there",
"these",
"those",
"your",
"into",
"such",
"been",
"where",
"when",
"which",
"using",
"also",
"than",
"could",
"would",
"should",
"might",
"cannot",
"however",
"therefore",
"thereof",
"hereby",
"herein",
"hereof",
"based",
"system",
"application",
"service",
"provide",
"provided",
"including",
"include",
"includes",
"version",
"update",
"updates",
"usage",
]
.into_iter()
.collect()
});
static TOKEN_RE: std::sync::LazyLock<Regex> =
std::sync::LazyLock::new(|| Regex::new(r"(?i)[a-z0-9][a-z0-9'-]+").unwrap());
static STOPWORDS: std::sync::LazyLock<BTreeSet<&'static str>> =
std::sync::LazyLock::new(|| {
[
"the",
"and",
"for",
"with",
"that",
"from",
"this",
"were",
"have",
"has",
"will",
"shall",
"into",
"about",
"without",
"within",
"between",
"because",
"over",
"under",
"after",
"before",
"until",
"while",
"their",
"there",
"these",
"those",
"your",
"into",
"such",
"been",
"where",
"when",
"which",
"using",
"also",
"than",
"could",
"would",
"should",
"might",
"cannot",
"however",
"therefore",
"thereof",
"hereby",
"herein",
"hereof",
"based",
"system",
"application",
"service",
"provide",
"provided",
"including",
"include",
"includes",
"version",
"update",
"updates",
"usage",
]
.into_iter()
.collect()
});
let mut counts: BTreeMap<String, u32> = BTreeMap::new();
for token in TOKEN_RE.find_iter(text) {
@@ -131,8 +134,9 @@ fn extract_keywords(text: &str, limit: usize) -> Vec<String> {
}
fn derive_labels(text: &str, limit: usize) -> Vec<String> {
static PHRASE_RE: Lazy<Regex> =
Lazy::new(|| Regex::new(r"(?m)^(?P<phrase>[A-Z][A-Za-z0-9 &/-]{3,})$").unwrap());
static PHRASE_RE: std::sync::LazyLock<Regex> = std::sync::LazyLock::new(|| {
Regex::new(r"(?m)^(?P<phrase>[A-Z][A-Za-z0-9 &/-]{3,})$").unwrap()
});
let mut labels = BTreeSet::new();
for caps in PHRASE_RE.captures_iter(text) {
@@ -166,19 +170,19 @@ fn extract_dates(text: &str) -> Vec<String> {
// 2. ISO dates: 2024-09-01
// 3. US format: 09/01/2024
// 4. Spelled out: September 1, 2024 or Sept 1, 2024 or 1 September 2024
static DATE_RE: Lazy<Regex> = Lazy::new(|| {
static DATE_RE: std::sync::LazyLock<Regex> = std::sync::LazyLock::new(|| {
Regex::new(r"(?i)\b((?:19|20)\d{2}|\d{4}-\d{2}-\d{2}|\d{2}/\d{2}/\d{4})\b").unwrap()
});
// Match spelled-out dates like "September 1, 2024", "Sept 10, 2024", "September 1st, 2024"
static SPELLED_DATE_RE: Lazy<Regex> = Lazy::new(|| {
static SPELLED_DATE_RE: std::sync::LazyLock<Regex> = std::sync::LazyLock::new(|| {
Regex::new(
r"(?i)\b((?:January|February|March|April|May|June|July|August|September|October|November|December|Jan|Feb|Mar|Apr|Jun|Jul|Aug|Sep|Sept|Oct|Nov|Dec)\.?\s+\d{1,2}(?:st|nd|rd|th)?,?\s+(?:19|20)\d{2})\b"
).unwrap()
});
// Match European format: "1 September 2024", "1st September 2024"
static EURO_DATE_RE: Lazy<Regex> = Lazy::new(|| {
static EURO_DATE_RE: std::sync::LazyLock<Regex> = std::sync::LazyLock::new(|| {
Regex::new(
r"(?i)\b(\d{1,2}(?:st|nd|rd|th)?\s+(?:January|February|March|April|May|June|July|August|September|October|November|December|Jan|Feb|Mar|Apr|Jun|Jul|Aug|Sep|Sept|Oct|Nov|Dec)\.?\s+(?:19|20)\d{2})\b"
).unwrap()
@@ -217,7 +221,7 @@ mod tests {
#[test]
fn produces_keywords_and_labels() {
let text = "Rust memory engines power efficient systems. Memory safety ensures reliability in 2025.";
let result = AutoTagger::default().analyse(text, true);
let result = AutoTagger.analyse(text, true);
assert!(result.tags.iter().any(|tag| tag.contains("memory")));
assert!(!result.content_dates.is_empty());
}
+11 -3
View File
@@ -1,3 +1,5 @@
// Safe expect: Static NER model lookup with guaranteed default.
#![allow(clippy::expect_used)]
//! Named Entity Recognition (NER) module using DistilBERT-NER ONNX.
//!
//! This module provides entity extraction capabilities using DistilBERT-NER,
@@ -10,7 +12,7 @@
//!
//! # Simple Interface
//!
//! Unlike GLiNER, DistilBERT-NER uses standard BERT tokenization:
//! Unlike `GLiNER`, DistilBERT-NER uses standard BERT tokenization:
//! - Input: `input_ids`, `attention_mask`
//! - Output: per-token logits for B-PER, I-PER, B-ORG, I-ORG, B-LOC, I-LOC, B-MISC, I-MISC, O
@@ -25,7 +27,7 @@ use std::path::{Path, PathBuf};
/// Model name for downloads and caching
pub const NER_MODEL_NAME: &str = "distilbert-ner";
/// Model download URL (HuggingFace)
/// Model download URL (`HuggingFace`)
pub const NER_MODEL_URL: &str =
"https://huggingface.co/dslim/distilbert-NER/resolve/main/onnx/model.onnx";
@@ -82,11 +84,13 @@ pub static NER_MODELS: &[NerModelInfo] = &[NerModelInfo {
}];
/// Get NER model info by name
#[must_use]
pub fn get_ner_model_info(name: &str) -> Option<&'static NerModelInfo> {
NER_MODELS.iter().find(|m| m.name == name)
}
/// Get default NER model info
#[must_use]
pub fn default_ner_model_info() -> &'static NerModelInfo {
NER_MODELS
.iter()
@@ -114,7 +118,8 @@ pub struct ExtractedEntity {
}
impl ExtractedEntity {
/// Convert the raw entity type to our EntityKind enum
/// Convert the raw entity type to our `EntityKind` enum
#[must_use]
pub fn to_entity_kind(&self) -> EntityKind {
match self.entity_type.to_uppercase().as_str() {
"PER" | "PERSON" | "B-PER" | "I-PER" => EntityKind::Person,
@@ -557,16 +562,19 @@ impl NerModel {
// ============================================================================
/// Get the expected path for the NER model in the models directory
#[must_use]
pub fn ner_model_path(models_dir: &Path) -> PathBuf {
models_dir.join(NER_MODEL_NAME).join("model.onnx")
}
/// Get the expected path for the NER tokenizer in the models directory
#[must_use]
pub fn ner_tokenizer_path(models_dir: &Path) -> PathBuf {
models_dir.join(NER_MODEL_NAME).join("tokenizer.json")
}
/// Check if NER model is installed
#[must_use]
pub fn is_ner_model_installed(models_dir: &Path) -> bool {
ner_model_path(models_dir).exists() && ner_tokenizer_path(models_dir).exists()
}
+596
View File
@@ -0,0 +1,596 @@
//! API-based embedding providers (OpenAI, Anthropic, etc.)
//!
//! This module provides cloud API embedding generation, enabling semantic search
//! using external embedding services. Requires the `api_embed` feature.
//!
//! # Example
//!
//! ```ignore
//! use memvid_core::api_embed::{OpenAIConfig, OpenAIEmbedder};
//! use memvid_core::types::embedding::EmbeddingProvider;
//!
//! // Requires OPENAI_API_KEY environment variable
//! let config = OpenAIConfig::default();
//! let embedder = OpenAIEmbedder::new(config)?;
//!
//! let embedding = embedder.embed_text("Hello, world!")?;
//! println!("Embedding dimension: {}", embedding.len());
//! ```
use crate::error::{MemvidError, Result};
use crate::types::embedding::EmbeddingProvider;
use reqwest::blocking::Client;
use reqwest::header::{AUTHORIZATION, CONTENT_TYPE, HeaderMap, HeaderValue};
use serde::{Deserialize, Serialize};
use std::time::Duration;
// ============================================================================
// OpenAI Models Registry
// ============================================================================
/// OpenAI embedding model information
#[derive(Debug, Clone)]
pub struct OpenAIModelInfo {
/// Model identifier (e.g., "text-embedding-3-small")
pub name: &'static str,
/// Output embedding dimension
pub dimension: usize,
/// Maximum input tokens
pub max_tokens: usize,
/// Maximum texts per batch request
pub max_batch_size: usize,
/// Whether this is the default model
pub is_default: bool,
}
/// Available OpenAI embedding models
pub static OPENAI_MODELS: &[OpenAIModelInfo] = &[
OpenAIModelInfo {
name: "text-embedding-3-small",
dimension: 1536,
max_tokens: 8191,
max_batch_size: 2048,
is_default: true,
},
OpenAIModelInfo {
name: "text-embedding-3-large",
dimension: 3072,
max_tokens: 8191,
max_batch_size: 2048,
is_default: false,
},
OpenAIModelInfo {
name: "text-embedding-ada-002",
dimension: 1536,
max_tokens: 8191,
max_batch_size: 2048,
is_default: false,
},
];
/// Get model info by name, defaults to text-embedding-3-small
#[must_use]
pub fn get_openai_model_info(name: &str) -> &'static OpenAIModelInfo {
OPENAI_MODELS
.iter()
.find(|m| m.name == name)
.unwrap_or_else(|| OPENAI_MODELS.iter().find(|m| m.is_default).unwrap())
}
/// Get the default model info
#[must_use]
pub fn default_openai_model_info() -> &'static OpenAIModelInfo {
OPENAI_MODELS.iter().find(|m| m.is_default).unwrap()
}
// ============================================================================
// Configuration
// ============================================================================
/// Configuration for OpenAI embedding provider
#[derive(Debug, Clone)]
pub struct OpenAIConfig {
/// Model name (e.g., "text-embedding-3-small")
pub model: String,
/// Environment variable name for API key (default: "OPENAI_API_KEY")
pub api_key_env: String,
/// Custom API base URL (for Azure OpenAI, proxies, etc.)
/// Default: "https://api.openai.com/v1"
pub base_url: String,
/// Request timeout in seconds
pub timeout_secs: u64,
/// Maximum retries on rate limit (429) errors
pub max_retries: u32,
/// Initial backoff in milliseconds for exponential retry
pub initial_backoff_ms: u64,
}
impl Default for OpenAIConfig {
fn default() -> Self {
Self {
model: "text-embedding-3-small".to_string(),
api_key_env: "OPENAI_API_KEY".to_string(),
base_url: "https://api.openai.com/v1".to_string(),
timeout_secs: 30,
max_retries: 3,
initial_backoff_ms: 1000,
}
}
}
impl OpenAIConfig {
/// Create config for text-embedding-3-small (default, fastest)
#[must_use]
pub fn small() -> Self {
Self::default()
}
/// Create config for text-embedding-3-large (highest quality)
#[must_use]
pub fn large() -> Self {
Self {
model: "text-embedding-3-large".to_string(),
..Default::default()
}
}
/// Create config for text-embedding-ada-002 (legacy)
#[must_use]
pub fn ada() -> Self {
Self {
model: "text-embedding-ada-002".to_string(),
..Default::default()
}
}
/// Set custom base URL (for Azure OpenAI or proxies)
#[must_use]
pub fn with_base_url(mut self, url: impl Into<String>) -> Self {
self.base_url = url.into();
self
}
/// Set custom API key environment variable name
#[must_use]
pub fn with_api_key_env(mut self, env_var: impl Into<String>) -> Self {
self.api_key_env = env_var.into();
self
}
/// Set request timeout
#[must_use]
pub fn with_timeout(mut self, secs: u64) -> Self {
self.timeout_secs = secs;
self
}
}
// ============================================================================
// API Request/Response Types
// ============================================================================
#[derive(Serialize)]
struct EmbeddingRequest<'a> {
model: &'a str,
input: Vec<&'a str>,
encoding_format: &'a str,
}
#[derive(Deserialize)]
struct EmbeddingResponse {
data: Vec<EmbeddingData>,
#[allow(dead_code)]
usage: Usage,
}
#[derive(Deserialize)]
struct EmbeddingData {
embedding: Vec<f32>,
#[allow(dead_code)]
index: usize,
}
#[derive(Deserialize)]
struct Usage {
#[allow(dead_code)]
prompt_tokens: usize,
#[allow(dead_code)]
total_tokens: usize,
}
#[derive(Deserialize)]
struct ApiError {
error: ApiErrorDetail,
}
#[derive(Deserialize)]
struct ApiErrorDetail {
message: String,
#[serde(rename = "type")]
error_type: Option<String>,
}
// ============================================================================
// OpenAI Embedder
// ============================================================================
/// OpenAI embedding provider
///
/// Generates embeddings using OpenAI's embedding API. Requires the `OPENAI_API_KEY`
/// environment variable to be set (or a custom env var via config).
///
/// # Example
///
/// ```ignore
/// use memvid_core::api_embed::{OpenAIConfig, OpenAIEmbedder};
/// use memvid_core::types::embedding::EmbeddingProvider;
///
/// let embedder = OpenAIEmbedder::new(OpenAIConfig::default())?;
/// let embedding = embedder.embed_text("Hello, world!")?;
/// ```
pub struct OpenAIEmbedder {
config: OpenAIConfig,
model_info: &'static OpenAIModelInfo,
client: Client,
api_key: String,
}
impl OpenAIEmbedder {
/// Create a new OpenAI embedder
///
/// Reads the API key from the environment variable specified in config.
/// Returns an error if the API key is not set.
pub fn new(config: OpenAIConfig) -> Result<Self> {
let api_key =
std::env::var(&config.api_key_env).map_err(|_| MemvidError::EmbeddingFailed {
reason: format!(
"API key not found. Set the {} environment variable.",
config.api_key_env
)
.into(),
})?;
if api_key.is_empty() {
return Err(MemvidError::EmbeddingFailed {
reason: format!("{} environment variable is empty", config.api_key_env).into(),
});
}
let model_info = get_openai_model_info(&config.model);
let client = Client::builder()
.timeout(Duration::from_secs(config.timeout_secs))
.build()
.map_err(|e| MemvidError::EmbeddingFailed {
reason: format!("Failed to create HTTP client: {}", e).into(),
})?;
tracing::info!(
model = %model_info.name,
dimension = model_info.dimension,
"OpenAI embedder initialized"
);
Ok(Self {
config,
model_info,
client,
api_key,
})
}
/// Get model info
#[must_use]
pub fn model_info(&self) -> &'static OpenAIModelInfo {
self.model_info
}
/// Make an embedding request with retry logic
fn request_embeddings(&self, texts: &[&str]) -> Result<Vec<Vec<f32>>> {
let url = format!("{}/embeddings", self.config.base_url);
let request_body = EmbeddingRequest {
model: &self.config.model,
input: texts.to_vec(),
encoding_format: "float",
};
let mut headers = HeaderMap::new();
headers.insert(CONTENT_TYPE, HeaderValue::from_static("application/json"));
headers.insert(
AUTHORIZATION,
HeaderValue::from_str(&format!("Bearer {}", self.api_key)).map_err(|_| {
MemvidError::EmbeddingFailed {
reason: "Invalid API key format".into(),
}
})?,
);
let mut backoff_ms = self.config.initial_backoff_ms;
let mut last_error = None;
for attempt in 0..=self.config.max_retries {
if attempt > 0 {
tracing::warn!(
attempt = attempt,
backoff_ms = backoff_ms,
"Retrying OpenAI request after rate limit"
);
std::thread::sleep(Duration::from_millis(backoff_ms));
backoff_ms *= 2; // Exponential backoff
}
let response = self
.client
.post(&url)
.headers(headers.clone())
.json(&request_body)
.send();
match response {
Ok(resp) => {
let status = resp.status();
if status.is_success() {
let embedding_response: EmbeddingResponse =
resp.json().map_err(|e| MemvidError::EmbeddingFailed {
reason: format!("Failed to parse response: {}", e).into(),
})?;
// Sort by index to ensure correct order
let mut data = embedding_response.data;
data.sort_by_key(|d| d.index);
let embeddings: Vec<Vec<f32>> =
data.into_iter().map(|d| d.embedding).collect();
tracing::debug!(
texts = texts.len(),
dimension = embeddings.first().map(|e| e.len()).unwrap_or(0),
"Generated OpenAI embeddings"
);
return Ok(embeddings);
}
// Handle rate limiting
if status.as_u16() == 429 {
last_error = Some(MemvidError::EmbeddingFailed {
reason: "Rate limit exceeded".into(),
});
continue;
}
// Parse error response
let error_text = resp.text().unwrap_or_default();
let error_msg =
if let Ok(api_error) = serde_json::from_str::<ApiError>(&error_text) {
format!(
"OpenAI API error ({}): {}",
api_error.error.error_type.unwrap_or_default(),
api_error.error.message
)
} else {
format!("OpenAI API error ({}): {}", status, error_text)
};
return Err(MemvidError::EmbeddingFailed {
reason: error_msg.into(),
});
}
Err(e) => {
// Network error - might be transient
last_error = Some(MemvidError::EmbeddingFailed {
reason: format!("Request failed: {}", e).into(),
});
if e.is_timeout() || e.is_connect() {
continue; // Retry on timeout or connection errors
}
return Err(last_error.unwrap());
}
}
}
// All retries exhausted
Err(last_error.unwrap_or_else(|| MemvidError::EmbeddingFailed {
reason: "Max retries exceeded".into(),
}))
}
}
impl std::fmt::Debug for OpenAIEmbedder {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
f.debug_struct("OpenAIEmbedder")
.field("config", &self.config)
.field("model_info", &self.model_info)
.field("api_key", &"[REDACTED]")
.finish()
}
}
// ============================================================================
// EmbeddingProvider Implementation
// ============================================================================
impl EmbeddingProvider for OpenAIEmbedder {
fn kind(&self) -> &str {
"openai"
}
fn model(&self) -> &str {
self.model_info.name
}
fn dimension(&self) -> usize {
self.model_info.dimension
}
fn embed_text(&self, text: &str) -> Result<Vec<f32>> {
let embeddings = self.request_embeddings(&[text])?;
embeddings
.into_iter()
.next()
.ok_or_else(|| MemvidError::EmbeddingFailed {
reason: "No embedding returned".into(),
})
}
fn embed_batch(&self, texts: &[&str]) -> Result<Vec<Vec<f32>>> {
if texts.is_empty() {
return Ok(Vec::new());
}
// Split into chunks respecting API batch size limit
let max_batch = self.model_info.max_batch_size;
let mut all_embeddings = Vec::with_capacity(texts.len());
for chunk in texts.chunks(max_batch) {
let chunk_embeddings = self.request_embeddings(chunk)?;
all_embeddings.extend(chunk_embeddings);
}
Ok(all_embeddings)
}
fn is_ready(&self) -> bool {
// We have an API key, so we're ready
!self.api_key.is_empty()
}
}
// ============================================================================
// Tests
// ============================================================================
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn test_model_registry() {
assert_eq!(OPENAI_MODELS.len(), 3);
let default = default_openai_model_info();
assert_eq!(default.name, "text-embedding-3-small");
assert_eq!(default.dimension, 1536);
assert!(default.is_default);
}
#[test]
fn test_get_model_info() {
let small = get_openai_model_info("text-embedding-3-small");
assert_eq!(small.dimension, 1536);
let large = get_openai_model_info("text-embedding-3-large");
assert_eq!(large.dimension, 3072);
let ada = get_openai_model_info("text-embedding-ada-002");
assert_eq!(ada.dimension, 1536);
// Unknown model should return default
let unknown = get_openai_model_info("unknown-model");
assert_eq!(unknown.name, "text-embedding-3-small");
}
#[test]
fn test_config_defaults() {
let config = OpenAIConfig::default();
assert_eq!(config.model, "text-embedding-3-small");
assert_eq!(config.api_key_env, "OPENAI_API_KEY");
assert_eq!(config.base_url, "https://api.openai.com/v1");
assert_eq!(config.timeout_secs, 30);
assert_eq!(config.max_retries, 3);
}
#[test]
fn test_config_builders() {
let small = OpenAIConfig::small();
assert_eq!(small.model, "text-embedding-3-small");
let large = OpenAIConfig::large();
assert_eq!(large.model, "text-embedding-3-large");
let ada = OpenAIConfig::ada();
assert_eq!(ada.model, "text-embedding-ada-002");
}
#[test]
fn test_config_with_methods() {
let config = OpenAIConfig::default()
.with_base_url("https://custom.api.com")
.with_api_key_env("MY_API_KEY")
.with_timeout(60);
assert_eq!(config.base_url, "https://custom.api.com");
assert_eq!(config.api_key_env, "MY_API_KEY");
assert_eq!(config.timeout_secs, 60);
}
#[test]
fn test_embedder_requires_api_key() {
// Use a custom env var name that doesn't exist
let config = OpenAIConfig::default().with_api_key_env("NONEXISTENT_API_KEY_12345");
let result = OpenAIEmbedder::new(config);
assert!(result.is_err());
let err_msg = format!("{:?}", result.unwrap_err());
assert!(err_msg.contains("NONEXISTENT_API_KEY_12345"));
}
#[test]
fn test_embedder_validates_config() {
// Test that config with custom env var works correctly
let config = OpenAIConfig::default()
.with_api_key_env("CUSTOM_KEY_VAR")
.with_base_url("https://custom.openai.com/v1");
assert_eq!(config.api_key_env, "CUSTOM_KEY_VAR");
assert_eq!(config.base_url, "https://custom.openai.com/v1");
// Creating embedder should fail since CUSTOM_KEY_VAR is not set
let result = OpenAIEmbedder::new(config);
assert!(result.is_err());
}
/// Integration test - requires OPENAI_API_KEY to be set
/// Run with: cargo test --features api_embed test_openai_integration -- --ignored
#[test]
#[ignore]
fn test_openai_integration() {
let config = OpenAIConfig::default();
let embedder = OpenAIEmbedder::new(config).expect("Failed to create embedder");
// Test single embedding
let embedding = embedder
.embed_text("Hello, world!")
.expect("Failed to embed text");
assert_eq!(embedding.len(), 1536);
// Test batch embedding
let texts = vec!["Hello", "World", "Test"];
let embeddings = embedder.embed_batch(&texts).expect("Failed to batch embed");
assert_eq!(embeddings.len(), 3);
assert!(embeddings.iter().all(|e| e.len() == 1536));
// Test EmbeddingProvider trait
assert_eq!(embedder.kind(), "openai");
assert_eq!(embedder.model(), "text-embedding-3-small");
assert_eq!(embedder.dimension(), 1536);
assert!(embedder.is_ready());
}
/// Integration test with text-embedding-3-large
#[test]
#[ignore]
fn test_openai_large_integration() {
let config = OpenAIConfig::large();
let embedder = OpenAIEmbedder::new(config).expect("Failed to create embedder");
let embedding = embedder
.embed_text("Test with large model")
.expect("Failed to embed text");
assert_eq!(embedding.len(), 3072);
}
}
+91 -8
View File
@@ -1,3 +1,5 @@
// Safe expect: Static CLIP model lookup with guaranteed default.
#![allow(clippy::unwrap_used, clippy::expect_used)]
//! CLIP (Contrastive Language-Image Pre-training) visual embeddings module.
//!
//! This module provides visual understanding capabilities using MobileCLIP-S2,
@@ -28,11 +30,69 @@ use std::time::Duration;
use crate::{MemvidError, Result, types::FrameId};
// ============================================================================
// Stderr Suppression for macOS
// ============================================================================
// ONNX Runtime on macOS emits "Context leak detected, msgtracer returned -1"
// warnings from Apple's tracing infrastructure. These are harmless but noisy.
#[cfg(all(feature = "clip", target_os = "macos"))]
mod stderr_suppress {
use std::fs::File;
use std::io;
use std::os::unix::io::{AsRawFd, RawFd};
pub struct StderrSuppressor {
original_stderr: RawFd,
#[allow(dead_code)]
dev_null: File,
}
impl StderrSuppressor {
pub fn new() -> io::Result<Self> {
let dev_null = File::open("/dev/null")?;
let original_stderr = unsafe { libc::dup(2) };
if original_stderr == -1 {
return Err(io::Error::last_os_error());
}
let result = unsafe { libc::dup2(dev_null.as_raw_fd(), 2) };
if result == -1 {
unsafe { libc::close(original_stderr) };
return Err(io::Error::last_os_error());
}
Ok(Self {
original_stderr,
dev_null,
})
}
}
impl Drop for StderrSuppressor {
fn drop(&mut self) {
unsafe {
libc::dup2(self.original_stderr, 2);
libc::close(self.original_stderr);
}
}
}
}
#[cfg(all(feature = "clip", not(target_os = "macos")))]
mod stderr_suppress {
pub struct StderrSuppressor;
impl StderrSuppressor {
pub fn new() -> std::io::Result<Self> {
Ok(Self)
}
}
}
// ============================================================================
// Configuration Constants
// ============================================================================
/// CLIP index decode limit (512MB max)
#[allow(clippy::cast_possible_truncation)]
const CLIP_DECODE_LIMIT: usize = crate::MAX_INDEX_BYTES as usize;
/// MobileCLIP-S2 embedding dimensions
@@ -44,7 +104,7 @@ pub const SIGLIP_DIMS: u32 = 768;
/// Default input resolution for MobileCLIP-S2
pub const MOBILECLIP_INPUT_SIZE: u32 = 256;
/// Default input resolution for SigLIP
/// Default input resolution for `SigLIP`
pub const SIGLIP_INPUT_SIZE: u32 = 224;
/// Minimum image dimension to process (skip icons, bullets)
@@ -73,7 +133,7 @@ fn clip_config() -> impl bincode::config::Config {
// Model Registry
// ============================================================================
/// Available CLIP models with verified HuggingFace URLs
/// Available CLIP models with verified `HuggingFace` URLs
#[derive(Debug, Clone)]
pub struct ClipModelInfo {
/// Model identifier
@@ -138,6 +198,7 @@ pub static CLIP_MODELS: &[ClipModelInfo] = &[
];
/// Get model info by name, defaults to mobileclip-s2
#[must_use]
pub fn get_model_info(name: &str) -> &'static ClipModelInfo {
CLIP_MODELS
.iter()
@@ -151,6 +212,7 @@ pub fn get_model_info(name: &str) -> &'static ClipModelInfo {
}
/// Get the default model info
#[must_use]
pub fn default_model_info() -> &'static ClipModelInfo {
CLIP_MODELS
.iter()
@@ -181,6 +243,7 @@ pub struct ClipIndexBuilder {
}
impl ClipIndexBuilder {
#[must_use]
pub fn new() -> Self {
Self::default()
}
@@ -205,8 +268,7 @@ impl ClipIndexBuilder {
let dimension = self
.documents
.first()
.map(|doc| doc.embedding.len() as u32)
.unwrap_or(0);
.map_or(0, |doc| u32::try_from(doc.embedding.len()).unwrap_or(0));
Ok(ClipIndexArtifact {
bytes,
@@ -224,7 +286,7 @@ pub struct ClipIndexArtifact {
pub bytes: Vec<u8>,
/// Number of vectors in the index
pub vector_count: u64,
/// Embedding dimension (512 for MobileCLIP, 768 for SigLIP)
/// Embedding dimension (512 for `MobileCLIP`, 768 for `SigLIP`)
pub dimension: u32,
/// Blake3 checksum of the bytes
pub checksum: [u8; 32],
@@ -236,8 +298,15 @@ pub struct ClipIndex {
documents: Vec<ClipDocument>,
}
impl Default for ClipIndex {
fn default() -> Self {
Self::new()
}
}
impl ClipIndex {
/// Create a new empty CLIP index
#[must_use]
pub fn new() -> Self {
Self {
documents: Vec::new(),
@@ -286,6 +355,7 @@ impl ClipIndex {
}
/// Search for similar embeddings using L2 distance
#[must_use]
pub fn search(&self, query: &[f32], limit: usize) -> Vec<ClipSearchHit> {
if query.is_empty() {
return Vec::new();
@@ -321,6 +391,7 @@ impl ClipIndex {
}
/// Get embedding for a specific frame
#[must_use]
pub fn embedding_for(&self, frame_id: FrameId) -> Option<&[f32]> {
self.documents
.iter()
@@ -334,11 +405,13 @@ impl ClipIndex {
}
/// Number of documents in the index
#[must_use]
pub fn len(&self) -> usize {
self.documents.len()
}
/// Check if index is empty
#[must_use]
pub fn is_empty(&self) -> bool {
self.documents.is_empty()
}
@@ -351,8 +424,7 @@ impl ClipIndex {
let dimension = self
.documents
.first()
.map(|doc| doc.embedding.len() as u32)
.unwrap_or(0);
.map_or(0, |doc| u32::try_from(doc.embedding.len()).unwrap_or(0));
Ok(ClipIndexArtifact {
bytes,
@@ -397,6 +469,7 @@ pub struct ImageInfo {
impl ImageInfo {
/// Check if this image should be processed for CLIP embedding
#[must_use]
pub fn should_embed(&self) -> bool {
// Skip tiny images (icons, bullets)
if self.width < MIN_IMAGE_DIM || self.height < MIN_IMAGE_DIM {
@@ -405,7 +478,7 @@ impl ImageInfo {
// Skip extreme aspect ratios (dividers, lines)
let aspect = self.width as f32 / self.height as f32;
if aspect > MAX_ASPECT_RATIO || aspect < (1.0 / MAX_ASPECT_RATIO) {
if !((1.0 / MAX_ASPECT_RATIO)..=MAX_ASPECT_RATIO).contains(&aspect) {
return false;
}
@@ -664,6 +737,9 @@ mod model {
tracing::debug!(path = %vision_path.display(), "Loading CLIP vision model");
// Suppress stderr during ONNX session creation (macOS emits harmless warnings)
let _stderr_guard = stderr_suppress::StderrSuppressor::new().ok();
let session = Session::builder()
.map_err(|e| ClipError::InferenceError {
cause: e.to_string(),
@@ -681,6 +757,8 @@ mod model {
cause: format!("Failed to load vision model: {}", e),
})?;
// _stderr_guard dropped here, restoring stderr
*session_guard = Some(session);
tracing::info!(model = %self.model_info.name, "CLIP vision model loaded");
@@ -702,6 +780,9 @@ mod model {
tracing::debug!(path = %text_path.display(), "Loading CLIP text model");
// Suppress stderr during ONNX session creation (macOS emits harmless warnings)
let _stderr_guard = stderr_suppress::StderrSuppressor::new().ok();
let session = Session::builder()
.map_err(|e| ClipError::InferenceError {
cause: e.to_string(),
@@ -719,6 +800,8 @@ mod model {
cause: format!("Failed to load text model: {}", e),
})?;
// _stderr_guard dropped here, restoring stderr
*session_guard = Some(session);
tracing::info!(model = %self.model_info.name, "CLIP text model loaded");
+6 -1
View File
@@ -24,7 +24,7 @@ pub const WAL_OFFSET: u64 = HEADER_SIZE as u64;
/// Minimal WAL size for empty/small memories (auto-grows on demand).
pub const WAL_SIZE_TINY: u64 = 64 * 1024;
/// WAL size tiers based on requested capacity (<100MB).
pub const WAL_SIZE_SMALL: u64 = 1 * 1024 * 1024;
pub const WAL_SIZE_SMALL: u64 = 1024 * 1024;
/// WAL size for memories under 1GB.
pub const WAL_SIZE_MEDIUM: u64 = 4 * 1024 * 1024;
/// WAL size for memories under 10GB.
@@ -35,3 +35,8 @@ pub const WAL_SIZE_XLARGE: u64 = 64 * 1024 * 1024;
pub const WAL_CHECKPOINT_THRESHOLD: f64 = 0.75;
/// Additional checkpoint every N transactions (PRD default).
pub const WAL_CHECKPOINT_PERIOD: u64 = 1_000;
/// Memvid's Ed25519 public key for verifying signed tickets.
/// This key is used to verify that tickets were issued by the official Memvid control plane.
/// The corresponding private key is held securely on the Memvid dashboard.
pub const MEMVID_TICKET_PUBKEY: &str = "DFKNhP/yO5i1b9aKL+aHeBaGunz9sMfOF736fzYws4Q=";
+40 -78
View File
@@ -3,14 +3,12 @@ use std::io::{Read, Seek, SeekFrom, Write};
use std::path::{Path, PathBuf};
use atomic_write_file::AtomicWriteFile;
use rand::RngCore;
use rand::rngs::OsRng;
use zeroize::Zeroize;
use crate::encryption::constants::{MV2E_MAGIC, MV2E_VERSION, NONCE_SIZE, SALT_SIZE};
use crate::encryption::crypto::{decrypt, derive_key, encrypt};
use crate::encryption::capsule_stream::{lock_file_stream, unlock_file_stream};
use crate::encryption::crypto::{decrypt, derive_key};
use crate::encryption::error::EncryptionError;
use crate::encryption::types::{CipherAlgorithm, KdfAlgorithm, Mv2eHeader};
use crate::encryption::types::Mv2eHeader;
/// Lock (encrypt) an `.mv2` file into a `.mv2e` capsule.
pub fn lock_file(
@@ -18,46 +16,7 @@ pub fn lock_file(
output: Option<&Path>,
password: &[u8],
) -> Result<PathBuf, EncryptionError> {
let input = input.as_ref();
validate_mv2_file(input)?;
let plaintext = std::fs::read(input).map_err(|source| EncryptionError::Io {
source,
path: Some(input.to_path_buf()),
})?;
let mut salt = [0u8; SALT_SIZE];
let mut nonce = [0u8; NONCE_SIZE];
OsRng.fill_bytes(&mut salt);
OsRng.fill_bytes(&mut nonce);
let mut key = derive_key(password, &salt)?;
let ciphertext = encrypt(&plaintext, &key, &nonce)?;
let header = Mv2eHeader {
magic: MV2E_MAGIC,
version: MV2E_VERSION,
kdf_algorithm: KdfAlgorithm::Argon2id,
cipher_algorithm: CipherAlgorithm::Aes256Gcm,
salt,
nonce,
original_size: plaintext.len() as u64,
reserved: [0u8; 4],
};
let output_path = output
.map(PathBuf::from)
.unwrap_or_else(|| input.with_extension("mv2e"));
write_atomic(&output_path, |writer| {
writer.write_all(&header.encode())?;
writer.write_all(&ciphertext)?;
Ok(())
})?;
key.zeroize();
Ok(output_path)
lock_file_stream(input, output, password)
}
/// Unlock (decrypt) an `.mv2e` capsule into an `.mv2` file.
@@ -81,6 +40,30 @@ pub fn unlock_file(
})?;
let header = Mv2eHeader::decode(&header_bytes)?;
if header.reserved[0] == 0x01 {
unlock_file_stream(input, output, password)
} else {
unlock_file_oneshot(input, output, password, header)
}
}
fn unlock_file_oneshot(
input: &Path,
output: Option<&Path>,
password: &[u8],
header: Mv2eHeader,
) -> Result<PathBuf, EncryptionError> {
let mut file = File::open(input).map_err(|source| EncryptionError::Io {
source,
path: Some(input.to_path_buf()),
})?;
file.seek(SeekFrom::Start(Mv2eHeader::SIZE as u64))
.map_err(|source| EncryptionError::Io {
source,
path: Some(input.to_path_buf()),
})?;
let mut ciphertext = Vec::new();
file.read_to_end(&mut ciphertext)
.map_err(|source| EncryptionError::Io {
@@ -105,7 +88,7 @@ pub fn unlock_file(
.map(PathBuf::from)
.unwrap_or_else(|| input.with_extension("mv2"));
write_atomic(&output_path, |writer| {
write_atomic(&output_path, |writer| -> Result<(), EncryptionError> {
writer.write_all(&plaintext)?;
Ok(())
})?;
@@ -113,7 +96,7 @@ pub fn unlock_file(
Ok(output_path)
}
fn validate_mv2_file(path: &Path) -> Result<(), EncryptionError> {
pub fn validate_mv2_file(path: &Path) -> Result<(), EncryptionError> {
let mut file = File::open(path).map_err(|source| EncryptionError::Io {
source,
path: Some(path.to_path_buf()),
@@ -143,42 +126,21 @@ fn validate_mv2_bytes(bytes: &[u8]) -> Result<(), EncryptionError> {
Ok(())
}
fn write_atomic<F>(path: &Path, write_fn: F) -> Result<(), EncryptionError>
pub fn write_atomic<F, E>(path: &Path, write_fn: F) -> Result<(), E>
where
F: FnOnce(&mut File) -> std::io::Result<()>,
F: FnOnce(&mut File) -> Result<(), E>,
E: From<std::io::Error>,
{
let mut options = AtomicWriteFile::options();
options.read(false);
let mut atomic = options.open(path).map_err(|source| EncryptionError::Io {
source,
path: Some(path.to_path_buf()),
})?;
let mut atomic = options.open(path)?;
let file = atomic.as_file_mut();
file.set_len(0).map_err(|source| EncryptionError::Io {
source,
path: Some(path.to_path_buf()),
})?;
file.seek(SeekFrom::Start(0))
.map_err(|source| EncryptionError::Io {
source,
path: Some(path.to_path_buf()),
})?;
write_fn(file).map_err(|source| EncryptionError::Io {
source,
path: Some(path.to_path_buf()),
})?;
file.flush().map_err(|source| EncryptionError::Io {
source,
path: Some(path.to_path_buf()),
})?;
file.sync_all().map_err(|source| EncryptionError::Io {
source,
path: Some(path.to_path_buf()),
})?;
atomic.commit().map_err(|source| EncryptionError::Io {
source,
path: Some(path.to_path_buf()),
})?;
file.set_len(0)?;
file.seek(SeekFrom::Start(0))?;
write_fn(file)?;
file.flush()?;
file.sync_all()?;
atomic.commit()?;
Ok(())
}
+138
View File
@@ -0,0 +1,138 @@
use rand::RngCore;
use rand::rngs::OsRng;
use std::fs::File;
use std::io::{BufReader, BufWriter, Read, Write};
use std::path::{Path, PathBuf};
use zeroize::Zeroize;
use crate::encryption::capsule::{validate_mv2_file, write_atomic};
use crate::encryption::constants::{MV2E_MAGIC, MV2E_VERSION, NONCE_SIZE, SALT_SIZE};
use crate::encryption::crypto::{decrypt, derive_key, encrypt};
use crate::encryption::error::EncryptionError;
use crate::encryption::types::{CipherAlgorithm, KdfAlgorithm, Mv2eHeader};
const CHUNK_SIZE: usize = 1024 * 1024;
// format: [header][len0][chunk0][len1][chunk1]...
// reserved[0] == 0x01 => streaming framed format
pub fn lock_file_stream(
input: impl AsRef<Path>,
output: Option<&Path>,
password: &[u8],
) -> Result<PathBuf, EncryptionError> {
let input = input.as_ref();
validate_mv2_file(input)?;
let metadata = std::fs::metadata(input)?;
let mut salt = [0u8; SALT_SIZE];
let mut base_nonce = [0u8; NONCE_SIZE];
OsRng.fill_bytes(&mut salt);
OsRng.fill_bytes(&mut base_nonce);
let mut key = derive_key(password, &salt)?;
let header = Mv2eHeader {
magic: MV2E_MAGIC,
version: MV2E_VERSION,
kdf_algorithm: KdfAlgorithm::Argon2id,
cipher_algorithm: CipherAlgorithm::Aes256Gcm,
salt,
nonce: base_nonce,
original_size: metadata.len(),
reserved: [0x01, 0, 0, 0],
};
let output_path = output
.map(PathBuf::from)
.unwrap_or_else(|| input.with_extension("mv2e"));
let input_file = File::open(input)?;
let mut reader = BufReader::new(input_file);
write_atomic(&output_path, |file| -> Result<(), EncryptionError> {
let mut writer = BufWriter::new(file);
writer.write_all(&header.encode())?;
let mut buffer = vec![0u8; CHUNK_SIZE];
let mut chunk_index: u64 = 0;
loop {
let n = reader.read(&mut buffer)?;
if n == 0 {
break;
}
let mut nonce = base_nonce;
nonce[NONCE_SIZE - 8..].copy_from_slice(&chunk_index.to_be_bytes());
let ciphertext = encrypt(&buffer[..n], &key, &nonce)?;
let chunk_len = ciphertext.len() as u32;
writer.write_all(&chunk_len.to_le_bytes())?;
writer.write_all(&ciphertext)?;
chunk_index += 1;
}
writer.flush()?;
Ok(())
})?;
key.zeroize();
Ok(output_path)
}
pub fn unlock_file_stream(
input: impl AsRef<Path>,
output: Option<&Path>,
password: &[u8],
) -> Result<PathBuf, EncryptionError> {
let input = input.as_ref();
let input_file = File::open(input)?;
let mut reader = BufReader::new(input_file);
let mut header_bytes = [0u8; Mv2eHeader::SIZE];
reader.read_exact(&mut header_bytes)?;
let header = Mv2eHeader::decode(&header_bytes)?;
let mut key = derive_key(password, &header.salt)?;
let output_path = output
.map(PathBuf::from)
.unwrap_or_else(|| input.with_extension("mv2"));
write_atomic(&output_path, |file| -> Result<(), EncryptionError> {
let mut writer = BufWriter::new(file);
let mut chunk_index: u64 = 0;
loop {
let mut len_bytes = [0u8; 4];
match reader.read_exact(&mut len_bytes) {
Ok(()) => {}
Err(e) if e.kind() == std::io::ErrorKind::UnexpectedEof => break,
Err(e) => return Err(e.into()),
}
let chunk_len = u32::from_le_bytes(len_bytes) as usize;
let mut ciphertext = vec![0u8; chunk_len];
reader.read_exact(&mut ciphertext)?;
let mut nonce = header.nonce;
nonce[NONCE_SIZE - 8..].copy_from_slice(&chunk_index.to_be_bytes());
let plaintext = decrypt(&ciphertext, &key, &nonce)?;
writer.write_all(&plaintext)?;
chunk_index += 1;
}
writer.flush()?;
Ok(())
})?;
key.zeroize();
Ok(output_path)
}
+6
View File
@@ -43,3 +43,9 @@ pub enum EncryptionError {
#[error("Corrupted decryption - output is not a valid MV2 file")]
CorruptedDecryption,
}
impl From<std::io::Error> for EncryptionError {
fn from(source: std::io::Error) -> Self {
EncryptionError::Io { source, path: None }
}
}
+2
View File
@@ -5,12 +5,14 @@
//! need encrypted-at-rest capsules.
mod capsule;
mod capsule_stream;
mod constants;
mod crypto;
mod error;
mod types;
pub use capsule::{lock_file, unlock_file};
pub use capsule_stream::{lock_file_stream, unlock_file_stream};
pub use constants::*;
pub use error::EncryptionError;
pub use types::{CipherAlgorithm, KdfAlgorithm, Mv2eHeader};
+4 -4
View File
@@ -12,7 +12,7 @@ use crate::types::{FrameId, MemoryCard};
pub struct EnrichmentContext {
/// The frame ID being processed.
pub frame_id: FrameId,
/// The frame's URI (e.g., "mv2://session-1/msg-5").
/// The frame's URI (e.g., "<mv2://session-1/msg-5>").
pub uri: String,
/// The frame's text content.
pub text: String,
@@ -136,7 +136,7 @@ pub trait EnrichmentEngine: Send + Sync {
/// Check if the engine is ready for processing.
///
/// Returns true if init() has been called and the engine is ready.
/// Returns true if `init()` has been called and the engine is ready.
fn is_ready(&self) -> bool {
true
}
@@ -149,10 +149,10 @@ mod tests {
struct TestEngine;
impl EnrichmentEngine for TestEngine {
fn kind(&self) -> &str {
fn kind(&self) -> &'static str {
"test"
}
fn version(&self) -> &str {
fn version(&self) -> &'static str {
"1.0.0"
}
fn enrich(&self, _ctx: &EnrichmentContext) -> EnrichmentResult {
+4 -4
View File
@@ -88,7 +88,7 @@ impl ExtractionRule {
}
let mut builder = MemoryCardBuilder::new()
.kind(self.kind.clone())
.kind(self.kind)
.entity(&entity)
.slot(&slot)
.value(&value)
@@ -96,7 +96,7 @@ impl ExtractionRule {
.engine("rules", "1.0.0");
if let Some(polarity) = &self.polarity {
builder = builder.polarity(polarity.clone());
builder = builder.polarity(*polarity);
}
// Build with a placeholder ID (will be assigned by MemoriesTrack)
@@ -112,7 +112,7 @@ impl ExtractionRule {
fn expand_captures(&self, template: &str, caps: &regex::Captures) -> String {
let mut result = template.to_string();
for i in 0..10 {
let placeholder = format!("${}", i);
let placeholder = format!("${i}");
if let Some(m) = caps.get(i) {
result = result.replace(&placeholder, m.as_str());
}
@@ -836,7 +836,7 @@ impl RulesEngine {
}
impl EnrichmentEngine for RulesEngine {
fn kind(&self) -> &str {
fn kind(&self) -> &'static str {
"rules"
}
+19 -15
View File
@@ -72,6 +72,7 @@ pub struct EnrichmentWorkerHandle {
impl EnrichmentWorkerHandle {
/// Create a new worker handle.
#[must_use]
pub fn new() -> Self {
Self {
stop_signal: Arc::new(AtomicBool::new(false)),
@@ -89,16 +90,19 @@ impl EnrichmentWorkerHandle {
}
/// Check if stop was requested.
#[must_use]
pub fn should_stop(&self) -> bool {
self.stop_signal.load(Ordering::SeqCst)
}
/// Check if worker is currently running.
#[must_use]
pub fn is_running(&self) -> bool {
self.is_running.load(Ordering::SeqCst)
}
/// Get current statistics.
#[must_use]
pub fn stats(&self) -> EnrichmentWorkerStats {
EnrichmentWorkerStats {
frames_processed: self.frames_processed.load(Ordering::Relaxed),
@@ -137,6 +141,7 @@ impl EnrichmentWorkerHandle {
}
/// Clone the handle for sharing with the worker thread.
#[must_use]
pub fn clone_handle(&self) -> Self {
Self {
stop_signal: Arc::clone(&self.stop_signal),
@@ -261,6 +266,7 @@ pub struct EnrichmentProcessor {
impl EnrichmentProcessor {
/// Create a new enrichment processor.
#[must_use]
pub fn new(config: EnrichmentWorkerConfig) -> Self {
Self { config }
}
@@ -300,13 +306,12 @@ impl EnrichmentProcessor {
};
// Read current frame state
let (text, is_skim, _needs_embedding) = match read_frame(task.frame_id) {
Some(data) => data,
None => {
result.error = Some("Frame not found".to_string());
result.elapsed_ms = start.elapsed().as_millis() as u64;
return result;
}
let (text, is_skim, _needs_embedding) = if let Some(data) = read_frame(task.frame_id) {
data
} else {
result.error = Some("Frame not found".to_string());
result.elapsed_ms = start.elapsed().as_millis().try_into().unwrap_or(u64::MAX);
return result;
};
// Re-extract if this was a skim
@@ -334,7 +339,7 @@ impl EnrichmentProcessor {
result.error = Some(format!("Index update failed: {err}"));
}
result.elapsed_ms = start.elapsed().as_millis() as u64;
result.elapsed_ms = start.elapsed().as_millis().try_into().unwrap_or(u64::MAX);
result
}
}
@@ -371,13 +376,12 @@ pub fn run_worker_loop<G, P, M, C>(
while !handle.should_stop() {
// Get next task
let task = match get_next_task() {
Some(task) => task,
None => {
// Queue is empty, wait and check again
std::thread::sleep(Duration::from_millis(config.task_delay_ms * 10));
continue;
}
let task = if let Some(task) = get_next_task() {
task
} else {
// Queue is empty, wait and check again
std::thread::sleep(Duration::from_millis(config.task_delay_ms * 10));
continue;
};
// Process the task
+5 -1
View File
@@ -64,7 +64,7 @@ pub enum MemvidError {
Lock(String),
#[error(transparent)]
Locked(#[from] LockedError),
Locked(#[from] Box<LockedError>),
#[error("Checksum mismatch while validating {context}")]
ChecksumMismatch { context: &'static str },
@@ -205,6 +205,9 @@ pub enum MemvidError {
#[error("Reranking failed: {reason}")]
RerankFailed { reason: Box<str> },
#[error("Model mismatch: Index is bound to '{expected}', but requested model was '{actual}'")]
ModelMismatch { expected: String, actual: String },
#[error("Invalid query: {reason}")]
InvalidQuery { reason: String },
@@ -224,6 +227,7 @@ impl From<std::io::Error> for MemvidError {
}
}
#[cfg(feature = "lex")]
impl From<tantivy::TantivyError> for MemvidError {
fn from(value: tantivy::TantivyError) -> Self {
Self::Tantivy {
+216 -17
View File
@@ -16,7 +16,7 @@ use serde_json::{Value, json};
#[cfg(feature = "extractous")]
use extractous::Extractor;
#[cfg(feature = "extractous")]
use std::collections::HashMap;
use std::collections::{HashMap, VecDeque};
#[cfg(feature = "extractous")]
use std::sync::{Mutex, OnceLock};
@@ -30,6 +30,7 @@ pub struct ExtractedDocument {
}
impl ExtractedDocument {
#[must_use]
pub fn empty() -> Self {
Self {
text: None,
@@ -52,6 +53,89 @@ impl Default for ProcessorConfig {
}
}
// ============================================================================
// Extraction Cache with LRU Eviction
// ============================================================================
/// Default capacity for extraction cache (number of documents)
#[cfg(feature = "extractous")]
const DEFAULT_EXTRACTION_CACHE_CAPACITY: usize = 100;
/// LRU cache for extracted documents to avoid re-extracting the same content.
///
/// This cache has a maximum capacity and evicts the least recently used entries
/// when full, following the same pattern as `EmbeddingCache` in `text_embed.rs`.
#[cfg(feature = "extractous")]
struct ExtractionCache {
/// Cache storage: document hash -> extracted document
cache: HashMap<blake3::Hash, ExtractedDocument>,
/// LRU queue: tracks access order (most recent at front)
lru_queue: VecDeque<blake3::Hash>,
/// Maximum capacity
capacity: usize,
/// Cache hit count
hits: usize,
/// Cache miss count
misses: usize,
}
#[cfg(feature = "extractous")]
impl ExtractionCache {
fn new(capacity: usize) -> Self {
Self {
cache: HashMap::with_capacity(capacity),
lru_queue: VecDeque::with_capacity(capacity),
capacity,
hits: 0,
misses: 0,
}
}
fn get(&mut self, key: &blake3::Hash) -> Option<ExtractedDocument> {
if let Some(document) = self.cache.get(key) {
// Move to front (most recently used)
self.lru_queue.retain(|k| k != key);
self.lru_queue.push_front(*key);
self.hits += 1;
Some(document.clone())
} else {
self.misses += 1;
None
}
}
fn insert(&mut self, key: blake3::Hash, value: ExtractedDocument) {
// Check if already exists
if self.cache.contains_key(&key) {
// Update and move to front
self.cache.insert(key, value);
self.lru_queue.retain(|k| *k != key);
self.lru_queue.push_front(key);
return;
}
// Evict if at capacity
if self.cache.len() >= self.capacity {
if let Some(oldest_key) = self.lru_queue.pop_back() {
self.cache.remove(&oldest_key);
tracing::debug!(
evicted_hash = ?oldest_key,
"Evicted oldest entry from extraction cache"
);
}
}
// Insert new entry
self.cache.insert(key, value);
self.lru_queue.push_front(key);
}
#[allow(dead_code)]
fn stats(&self) -> (usize, usize, usize) {
(self.hits, self.misses, self.cache.len())
}
}
// ============================================================================
// DocumentProcessor - only available with extractous feature
// ============================================================================
@@ -71,8 +155,7 @@ impl Default for DocumentProcessor {
}
#[cfg(feature = "extractous")]
static EXTRACTION_CACHE: OnceLock<Mutex<HashMap<blake3::Hash, ExtractedDocument>>> =
OnceLock::new();
static EXTRACTION_CACHE: OnceLock<Mutex<ExtractionCache>> = OnceLock::new();
#[cfg(feature = "extractous")]
impl DocumentProcessor {
@@ -284,6 +367,7 @@ impl Default for DocumentProcessor {
#[cfg(not(feature = "extractous"))]
impl DocumentProcessor {
#[must_use]
pub fn new(config: ProcessorConfig) -> Self {
Self {
max_length: config.max_text_chars,
@@ -536,13 +620,15 @@ fn value_to_mime(value: &Value) -> Option<String> {
#[cfg(feature = "extractous")]
fn cache_lookup(hash: &blake3::Hash) -> Option<ExtractedDocument> {
let cache = EXTRACTION_CACHE.get_or_init(|| Mutex::new(HashMap::new()));
cache.lock().ok().and_then(|map| map.get(hash).cloned())
let cache = EXTRACTION_CACHE
.get_or_init(|| Mutex::new(ExtractionCache::new(DEFAULT_EXTRACTION_CACHE_CAPACITY)));
cache.lock().ok().and_then(|mut map| map.get(hash))
}
#[cfg(feature = "extractous")]
fn cache_store(hash: blake3::Hash, document: &ExtractedDocument) {
let cache = EXTRACTION_CACHE.get_or_init(|| Mutex::new(HashMap::new()));
let cache = EXTRACTION_CACHE
.get_or_init(|| Mutex::new(ExtractionCache::new(DEFAULT_EXTRACTION_CACHE_CAPACITY)));
if let Ok(mut map) = cache.lock() {
map.insert(hash, document.clone());
}
@@ -559,8 +645,8 @@ const PDF_LOPDF_MAX_BYTES: usize = 64 * 1024 * 1024; // 64 MiB hard cap
const PDF_LOPDF_MAX_PAGES: usize = 4_096;
/// Try multiple PDF extractors and return the best result
/// Returns (text, extractor_name) or None if no text found
/// Priority: pdf_oxide (2025, best accuracy) > pdf_extract > lopdf
/// Returns (text, `extractor_name`) or None if no text found
/// Priority: `pdf_oxide` (2025, best accuracy) > `pdf_extract` > lopdf
#[allow(dead_code)]
fn pdf_text_extract_best(bytes: &[u8]) -> Result<Option<(String, &'static str)>> {
let mut best_text: Option<String> = None;
@@ -631,7 +717,7 @@ fn pdf_text_extract_best(bytes: &[u8]) -> Result<Option<(String, &'static str)>>
// Use if better than current best
if best_text
.as_ref()
.map_or(true, |prev| trimmed.len() > prev.len())
.is_none_or(|prev| trimmed.len() > prev.len())
{
best_text = Some(trimmed.to_string());
best_source = "pdf_extract";
@@ -652,7 +738,7 @@ fn pdf_text_extract_best(bytes: &[u8]) -> Result<Option<(String, &'static str)>>
// Use lopdf result if better than previous
if best_text
.as_ref()
.map_or(true, |prev| trimmed.len() > prev.len())
.is_none_or(|prev| trimmed.len() > prev.len())
{
tracing::debug!(
target: "memvid::extract",
@@ -788,16 +874,14 @@ fn pdf_text_extract_lopdf(bytes: &[u8]) -> Result<Option<String>> {
})?;
// Try to decrypt if encrypted (empty password for unprotected PDFs)
if document.is_encrypted() {
if document.decrypt("").is_err() {
return Err(MemvidError::ExtractionFailed {
reason: "cannot decrypt password-protected PDF".into(),
});
}
if document.is_encrypted() && document.decrypt("").is_err() {
return Err(MemvidError::ExtractionFailed {
reason: "cannot decrypt password-protected PDF".into(),
});
}
// Decompress streams for better text extraction
let _ = document.decompress();
let () = document.decompress();
let mut page_numbers: Vec<u32> = document.get_pages().keys().copied().collect();
if page_numbers.is_empty() {
@@ -885,3 +969,118 @@ mod pdf_fix_tests {
assert!(!looks_like_pdf_structure_dump(normal_text));
}
}
// ============================================================================
// Tests for ExtractionCache LRU eviction
// ============================================================================
#[cfg(all(test, feature = "extractous"))]
mod extraction_cache_tests {
use super::*;
fn make_doc(content: &str) -> ExtractedDocument {
ExtractedDocument {
text: Some(content.to_string()),
metadata: serde_json::json!({}),
mime_type: Some("text/plain".to_string()),
}
}
#[test]
fn test_extraction_cache_basic() {
let mut cache = ExtractionCache::new(10);
let hash = blake3::hash(b"test document");
let doc = make_doc("test content");
cache.insert(hash, doc.clone());
let retrieved = cache.get(&hash);
assert!(retrieved.is_some());
assert_eq!(retrieved.unwrap().text, Some("test content".to_string()));
}
#[test]
fn test_extraction_cache_stats() {
let mut cache = ExtractionCache::new(10);
let hash = blake3::hash(b"test");
cache.insert(hash, make_doc("test"));
// Hit
let _ = cache.get(&hash);
// Miss
let missing = blake3::hash(b"missing");
let _ = cache.get(&missing);
let (hits, misses, size) = cache.stats();
assert_eq!(hits, 1);
assert_eq!(misses, 1);
assert_eq!(size, 1);
}
#[test]
fn test_extraction_cache_eviction() {
let mut cache = ExtractionCache::new(3);
// Insert 4 items, first should be evicted
for i in 0..4u8 {
let hash = blake3::hash(&[i]);
cache.insert(hash, make_doc(&format!("doc{}", i)));
}
// First item should be evicted
let evicted = blake3::hash(&[0u8]);
assert!(cache.get(&evicted).is_none());
// Last 3 should still exist
for i in 1..4u8 {
let hash = blake3::hash(&[i]);
assert!(cache.get(&hash).is_some());
}
}
#[test]
fn test_extraction_cache_lru_promotion() {
let mut cache = ExtractionCache::new(3);
// Insert 3 items: 0, 1, 2
for i in 0..3u8 {
let hash = blake3::hash(&[i]);
cache.insert(hash, make_doc(&format!("doc{}", i)));
}
// Access first item (promotes it to front)
let first = blake3::hash(&[0u8]);
let _ = cache.get(&first);
// Insert 4th item - should evict second (index 1, now oldest)
let new_hash = blake3::hash(&[3u8]);
cache.insert(new_hash, make_doc("doc3"));
// First should still exist (was accessed, got promoted)
assert!(cache.get(&first).is_some());
// Second should be evicted (was oldest after first was promoted)
let second = blake3::hash(&[1u8]);
assert!(cache.get(&second).is_none());
// Third and fourth should exist
let third = blake3::hash(&[2u8]);
assert!(cache.get(&third).is_some());
assert!(cache.get(&new_hash).is_some());
}
#[test]
fn test_extraction_cache_update_existing() {
let mut cache = ExtractionCache::new(3);
let hash = blake3::hash(b"test");
cache.insert(hash, make_doc("original"));
cache.insert(hash, make_doc("updated"));
let retrieved = cache.get(&hash);
assert_eq!(retrieved.unwrap().text, Some("updated".to_string()));
// Size should still be 1
let (_, _, size) = cache.stats();
assert_eq!(size, 1);
}
}
+18 -16
View File
@@ -43,6 +43,7 @@ impl Default for ExtractionBudget {
impl ExtractionBudget {
/// Create a budget with custom milliseconds.
#[must_use]
pub fn with_ms(ms: u64) -> Self {
Self {
budget: Duration::from_millis(ms),
@@ -51,6 +52,7 @@ impl ExtractionBudget {
}
/// Create an unlimited budget (extract everything).
#[must_use]
pub fn unlimited() -> Self {
Self {
budget: Duration::from_secs(3600), // 1 hour = effectively unlimited
@@ -79,11 +81,13 @@ pub struct BudgetedExtractionResult {
impl BudgetedExtractionResult {
/// Check if we got meaningful content.
#[must_use]
pub fn has_content(&self) -> bool {
!self.text.trim().is_empty()
}
/// Check if this is a skim (partial) extraction.
#[must_use]
pub fn is_skim(&self) -> bool {
!self.completed && self.sections_extracted < self.sections_total
}
@@ -134,7 +138,7 @@ pub fn extract_pdf_budgeted(
sections_extracted: estimated_pages,
sections_total: estimated_pages,
completed,
elapsed_ms: start.elapsed().as_millis() as u64,
elapsed_ms: start.elapsed().as_millis().try_into().unwrap_or(u64::MAX),
coverage: 1.0,
});
}
@@ -176,7 +180,7 @@ pub fn extract_pdf_budgeted(
sections_extracted: estimated_pages,
sections_total: estimated_pages,
completed,
elapsed_ms: start.elapsed().as_millis() as u64,
elapsed_ms: start.elapsed().as_millis().try_into().unwrap_or(u64::MAX),
coverage: 1.0,
});
}
@@ -229,16 +233,14 @@ fn extract_pdf_budgeted_lopdf(
})?;
// Handle encryption
if document.is_encrypted() {
if document.decrypt("").is_err() {
return Err(MemvidError::ExtractionFailed {
reason: "cannot decrypt password-protected PDF".into(),
});
}
if document.is_encrypted() && document.decrypt("").is_err() {
return Err(MemvidError::ExtractionFailed {
reason: "cannot decrypt password-protected PDF".into(),
});
}
// Decompress for better extraction
let _ = document.decompress();
let () = document.decompress();
// Get page numbers
let mut page_numbers: Vec<u32> = document.get_pages().keys().copied().collect();
@@ -248,7 +250,7 @@ fn extract_pdf_budgeted_lopdf(
sections_extracted: 0,
sections_total: 0,
completed: true,
elapsed_ms: start.elapsed().as_millis() as u64,
elapsed_ms: start.elapsed().as_millis().try_into().unwrap_or(u64::MAX),
coverage: 1.0,
});
}
@@ -357,7 +359,7 @@ pub fn extract_text_budgeted(
sections_extracted: sections,
sections_total: sections,
completed: true,
elapsed_ms: start.elapsed().as_millis() as u64,
elapsed_ms: start.elapsed().as_millis().try_into().unwrap_or(u64::MAX),
coverage: 1.0,
})
}
@@ -385,7 +387,7 @@ fn extract_ooxml_budgeted(
let start = Instant::now();
// Determine the document format from MIME first, then fall back to extension
let format = match mime.map(|m| m.to_lowercase()).as_deref() {
let format = match mime.map(str::to_lowercase).as_deref() {
Some(m) if m.contains("spreadsheetml") => Some(DocumentFormat::Xlsx),
Some(m) if m.contains("wordprocessingml") => Some(DocumentFormat::Docx),
Some(m) if m.contains("presentationml") => Some(DocumentFormat::Pptx),
@@ -426,7 +428,7 @@ fn extract_ooxml_budgeted(
sections_extracted: sections,
sections_total: sections,
completed: true,
elapsed_ms: start.elapsed().as_millis() as u64,
elapsed_ms: start.elapsed().as_millis().try_into().unwrap_or(u64::MAX),
coverage: 1.0,
})
}
@@ -439,7 +441,7 @@ fn extract_ooxml_budgeted(
sections_extracted: 0,
sections_total: 0,
completed: true,
elapsed_ms: start.elapsed().as_millis() as u64,
elapsed_ms: start.elapsed().as_millis().try_into().unwrap_or(u64::MAX),
coverage: 1.0,
})
}
@@ -453,7 +455,7 @@ pub fn extract_with_budget(
budget: ExtractionBudget,
) -> Result<BudgetedExtractionResult> {
// Check if PDF
let is_pdf = mime.map_or(false, |m| m.contains("pdf")) || is_pdf_magic(bytes);
let is_pdf = mime.is_some_and(|m| m.contains("pdf")) || is_pdf_magic(bytes);
if is_pdf {
extract_pdf_budgeted(bytes, budget)
@@ -585,7 +587,7 @@ fn finish_extraction(
sections_extracted,
sections_total: total_pages,
completed,
elapsed_ms: start.elapsed().as_millis() as u64,
elapsed_ms: start.elapsed().as_millis().try_into().unwrap_or(u64::MAX),
coverage,
})
}
+1 -1
View File
@@ -91,7 +91,7 @@ pub fn find_last_valid_footer(bytes: &[u8]) -> Option<FooterSlice<'_>> {
let candidate = &bytes[pos..pos + FOOTER_SIZE];
if let Some(footer) = CommitFooter::decode(candidate) {
let toc_end = pos;
let toc_len = footer.toc_len as usize;
let toc_len = usize::try_from(footer.toc_len).unwrap_or(0);
if toc_len == 0 || toc_len > toc_end {
search_end = pos;
continue;
+8 -2
View File
@@ -2,7 +2,7 @@
//!
//! This module provides hybrid retrieval that can:
//! 1. Parse natural language queries for relational patterns
//! 2. Match patterns against entity state (MemoryCards) or graph (Logic-Mesh)
//! 2. Match patterns against entity state (`MemoryCards`) or graph (Logic-Mesh)
//! 3. Combine graph-filtered candidates with vector ranking
use std::collections::{HashMap, HashSet};
@@ -186,18 +186,20 @@ fn extract_possessive_query(query: &str) -> Option<(String, String)> {
None
}
/// Graph matcher that executes patterns against MemoryCards.
/// Graph matcher that executes patterns against `MemoryCards`.
pub struct GraphMatcher<'a> {
memvid: &'a Memvid,
}
impl<'a> GraphMatcher<'a> {
/// Create a new graph matcher.
#[must_use]
pub fn new(memvid: &'a Memvid) -> Self {
Self { memvid }
}
/// Execute a graph pattern and return matching results.
#[must_use]
pub fn execute(&self, pattern: &GraphPattern) -> Vec<GraphMatchResult> {
let mut results = Vec::new();
@@ -325,6 +327,8 @@ pub fn hybrid_search(memvid: &mut Memvid, plan: &QueryPlan) -> Result<Vec<Hybrid
as_of_frame: None,
as_of_ts: None,
no_sketch: false,
acl_context: None,
acl_enforcement_mode: crate::types::AclEnforcementMode::Audit,
};
let response = memvid.search(request)?;
Ok(response
@@ -389,6 +393,8 @@ pub fn hybrid_search(memvid: &mut Memvid, plan: &QueryPlan) -> Result<Vec<Hybrid
as_of_frame: None,
as_of_ts: None,
no_sketch: false,
acl_context: None,
acl_enforcement_mode: crate::types::AclEnforcementMode::Audit,
};
let response = memvid.search(request)?;
return Ok(response
+22 -30
View File
@@ -92,18 +92,16 @@ impl HeaderCodec {
/// Decodes the canonical header bytes into a strongly typed struct after validation.
pub fn decode(bytes: &[u8; HEADER_SIZE]) -> Result<Header> {
let magic = bytes[..MAGIC.len()].try_into().unwrap();
// Extract fixed-size arrays from the header buffer
// All indices are compile-time constants, so these slices are guaranteed to fit
let magic: [u8; 4] = extract_array(bytes, 0)?;
if magic != MAGIC {
return Err(MemvidError::InvalidHeader {
reason: "magic mismatch".into(),
});
}
let version = u16::from_le_bytes(
bytes[VERSION_OFFSET..VERSION_OFFSET + 2]
.try_into()
.unwrap(),
);
let version = u16::from_le_bytes(extract_array(bytes, VERSION_OFFSET)?);
if version != EXPECTED_VERSION {
return Err(MemvidError::InvalidHeader {
reason: "unsupported version".into(),
@@ -116,40 +114,22 @@ impl HeaderCodec {
});
}
let footer_offset = u64::from_le_bytes(
bytes[FOOTER_OFFSET_POS..FOOTER_OFFSET_POS + 8]
.try_into()
.unwrap(),
);
let wal_offset = u64::from_le_bytes(
bytes[WAL_OFFSET_POS..WAL_OFFSET_POS + 8]
.try_into()
.unwrap(),
);
let footer_offset = u64::from_le_bytes(extract_array(bytes, FOOTER_OFFSET_POS)?);
let wal_offset = u64::from_le_bytes(extract_array(bytes, WAL_OFFSET_POS)?);
if wal_offset < WAL_OFFSET {
return Err(MemvidError::InvalidHeader {
reason: "wal_offset precedes data region".into(),
});
}
let wal_size =
u64::from_le_bytes(bytes[WAL_SIZE_POS..WAL_SIZE_POS + 8].try_into().unwrap());
let wal_size = u64::from_le_bytes(extract_array(bytes, WAL_SIZE_POS)?);
if wal_size == 0 {
return Err(MemvidError::InvalidHeader {
reason: "wal_size must be non-zero".into(),
});
}
let wal_checkpoint_pos = u64::from_le_bytes(
bytes[WAL_CHECKPOINT_POS..WAL_CHECKPOINT_POS + 8]
.try_into()
.unwrap(),
);
let wal_sequence = u64::from_le_bytes(
bytes[WAL_SEQUENCE_POS..WAL_SEQUENCE_POS + 8]
.try_into()
.unwrap(),
);
let mut toc_checksum = [0u8; 32];
toc_checksum.copy_from_slice(&bytes[TOC_CHECKSUM_POS..TOC_CHECKSUM_END]);
let wal_checkpoint_pos = u64::from_le_bytes(extract_array(bytes, WAL_CHECKPOINT_POS)?);
let wal_sequence = u64::from_le_bytes(extract_array(bytes, WAL_SEQUENCE_POS)?);
let toc_checksum: [u8; 32] = extract_array(bytes, TOC_CHECKSUM_POS)?;
Ok(Header {
magic,
@@ -164,6 +144,18 @@ impl HeaderCodec {
}
}
/// Extracts a fixed-size array from a byte slice at the given offset.
/// Returns an error if the slice is too short (should never happen with valid headers).
#[inline]
fn extract_array<const N: usize>(bytes: &[u8], offset: usize) -> Result<[u8; N]> {
bytes
.get(offset..offset + N)
.and_then(|s| s.try_into().ok())
.ok_or_else(|| MemvidError::InvalidHeader {
reason: "header truncated".into(),
})
}
fn clear_legacy_lock_metadata(buf: &mut [u8; HEADER_SIZE]) -> bool {
let region = &mut buf[LEGACY_LOCK_REGION_START..LEGACY_LOCK_REGION_END];
if region.iter().any(|byte| *byte != 0) {
+4 -1
View File
@@ -101,7 +101,9 @@ impl ManifestWal {
let checksum = hash(&payload);
self.file.seek(SeekFrom::Start(self.write_offset))?;
self.file.write_all(&(payload.len() as u32).to_le_bytes())?;
// Safe: validated payload.len() <= MAX_RECORD_BYTES (4MB) on line 96
self.file
.write_all(&(u32::try_from(payload.len()).unwrap_or(u32::MAX)).to_le_bytes())?;
self.file.write_all(checksum.as_bytes())?;
self.file.write_all(&payload)?;
@@ -193,6 +195,7 @@ impl ManifestWal {
break;
}
// Safe: validated payload_len <= MAX_RECORD_BYTES (4MB) on line 184
let mut payload = vec![0u8; payload_len as usize];
if let Err(err) = self.file.read_exact(&mut payload) {
if err.kind() == ErrorKind::UnexpectedEof {
+4
View File
@@ -298,6 +298,8 @@ pub fn read_track<R: Read + Seek>(
});
}
// Safe: length validated against MAX_TEMPORAL_TRACK_BYTES (16 GiB) on line 247
// and HEADER_SIZE is constant, so result fits in usize
let mut body = vec![0u8; (length - HEADER_SIZE as u64) as usize];
reader.read_exact(&mut body)?;
@@ -313,9 +315,11 @@ pub fn read_track<R: Read + Seek>(
});
}
// Safe: counts validated by checked_mul and total_expected == length check above
let mut mentions = Vec::with_capacity(entry_count as usize);
let mut anchors = Vec::with_capacity(anchor_count as usize);
// Safe: validated by checked_mul overflow check on line 283
let mentions_bytes = expected_entries_bytes as usize;
for chunk in body[..mentions_bytes].chunks_exact(MENTION_RECORD_SIZE) {
let raw = RawMention::decode(chunk)?;
+7 -3
View File
@@ -15,6 +15,7 @@ pub struct TimeIndexEntry {
}
impl TimeIndexEntry {
#[must_use]
pub fn new(timestamp: i64, frame_id: u64) -> Self {
Self {
timestamp,
@@ -31,7 +32,7 @@ pub fn append_track<W: Write + Seek>(
) -> Result<(u64, u64, [u8; 32])> {
entries.sort_by_key(|entry| (entry.timestamp, entry.frame_id));
let offset = writer.seek(SeekFrom::Current(0))?;
let offset = writer.stream_position()?;
let mut hasher = Hasher::new();
writer.write_all(&TIME_INDEX_MAGIC)?;
@@ -51,7 +52,7 @@ pub fn append_track<W: Write + Seek>(
hasher.update(&id_bytes);
}
let end = writer.seek(SeekFrom::Current(0))?;
let end = writer.stream_position()?;
let length = end - offset;
Ok((offset, length, *hasher.finalize().as_bytes()))
}
@@ -94,6 +95,8 @@ pub fn read_track<R: Read + Seek>(
});
}
// Safe: count validated by checked_mul and payload_bytes comparison above
#[allow(clippy::cast_possible_truncation)]
let mut entries = Vec::with_capacity(count as usize);
let mut prev: Option<TimeIndexEntry> = None;
for _ in 0..count {
@@ -126,6 +129,7 @@ pub fn read_track<R: Read + Seek>(
}
/// Calculates the checksum for the provided entries in canonical order.
#[must_use]
pub fn calculate_checksum(entries: &[TimeIndexEntry]) -> [u8; 32] {
let mut sorted = entries.to_vec();
sorted.sort_by_key(|entry| (entry.timestamp, entry.frame_id));
@@ -133,7 +137,7 @@ pub fn calculate_checksum(entries: &[TimeIndexEntry]) -> [u8; 32] {
let mut hasher = Hasher::new();
hasher.update(&TIME_INDEX_MAGIC);
hasher.update(&(sorted.len() as u64).to_le_bytes());
for entry in sorted.iter() {
for entry in &sorted {
hasher.update(&entry.timestamp.to_le_bytes());
hasher.update(&entry.frame_id.to_le_bytes());
}
+58 -18
View File
@@ -36,6 +36,7 @@ pub struct EmbeddedWal {
checkpoint_sequence: u64,
appends_since_checkpoint: u64,
read_only: bool,
skip_sync: bool,
}
impl EmbeddedWal {
@@ -67,8 +68,7 @@ impl EmbeddedWal {
.sum();
let sequence = entries
.last()
.map(|entry| entry.sequence)
.unwrap_or(checkpoint_sequence);
.map_or(checkpoint_sequence, |entry| entry.sequence);
let mut wal = Self {
file: clone,
@@ -81,6 +81,7 @@ impl EmbeddedWal {
checkpoint_sequence,
appends_since_checkpoint: 0,
read_only,
skip_sync: false,
};
if !wal.read_only {
@@ -153,6 +154,7 @@ impl EmbeddedWal {
Ok(self.sequence)
}
#[must_use]
pub fn should_checkpoint(&self) -> bool {
if self.read_only || self.region_size == 0 {
return false;
@@ -181,10 +183,7 @@ impl EmbeddedWal {
let (entries, next_head) =
Self::scan_records(&mut self.file, self.region_offset, self.region_size)?;
self.sequence = entries
.last()
.map(|entry| entry.sequence)
.unwrap_or(self.sequence);
self.sequence = entries.last().map_or(self.sequence, |entry| entry.sequence);
self.pending_bytes = entries
.iter()
.filter(|entry| entry.sequence > self.checkpoint_sequence)
@@ -205,6 +204,7 @@ impl EmbeddedWal {
.collect())
}
#[must_use]
pub fn stats(&self) -> WalStats {
WalStats {
region_size: self.region_size,
@@ -214,14 +214,33 @@ impl EmbeddedWal {
}
}
#[must_use]
pub fn region_offset(&self) -> u64 {
self.region_offset
}
#[must_use]
pub fn file(&self) -> &File {
&self.file
}
/// Enable or disable per-entry fsync.
///
/// When `skip` is `true`, `write_record()` will not call `sync_all()` after
/// each WAL append. The caller **must** call [`flush()`](Self::flush) after
/// the batch to ensure durability.
pub fn set_skip_sync(&mut self, skip: bool) {
self.skip_sync = skip;
}
/// Force an `fsync` on the underlying WAL file.
///
/// Call this after a batch of appends performed with `skip_sync = true`
/// to ensure all data is durable on disk.
pub fn flush(&mut self) -> Result<()> {
self.file.sync_all().map_err(Into::into)
}
fn initialise_sentinel(&mut self) -> Result<()> {
self.maybe_write_sentinel()
}
@@ -231,7 +250,8 @@ impl EmbeddedWal {
let digest = blake3::hash(payload);
let mut header = [0u8; ENTRY_HEADER_SIZE];
header[..8].copy_from_slice(&sequence.to_le_bytes());
header[8..12].copy_from_slice(&(payload.len() as u32).to_le_bytes());
header[8..12]
.copy_from_slice(&(u32::try_from(payload.len()).unwrap_or(u32::MAX)).to_le_bytes());
header[16..48].copy_from_slice(digest.as_bytes());
// Atomic write: combine header and payload into single buffer
@@ -249,7 +269,10 @@ impl EmbeddedWal {
// Force fsync to ensure data is durable before returning
// Critical for preventing corruption during rapid file operations
self.file.sync_all()?;
// In batch mode (skip_sync=true), fsync is deferred to flush() for performance
if !self.skip_sync {
self.file.sync_all()?;
}
Ok(())
}
@@ -263,6 +286,8 @@ impl EmbeddedWal {
let remaining = self.region_size - pos;
if remaining < ENTRY_HEADER_SIZE as u64 {
if remaining > 0 {
// Safe: remaining < ENTRY_HEADER_SIZE (48) so always fits in usize
#[allow(clippy::cast_possible_truncation)]
let zero_tail = vec![0u8; remaining as usize];
self.seek_and_write(pos, &zero_tail)?;
}
@@ -309,12 +334,12 @@ impl EmbeddedWal {
reason: "invalid wal sequence header".into(),
}
})?);
let length = u32::from_le_bytes(header[8..12].try_into().map_err(|_| {
MemvidError::WalCorruption {
let length = u64::from(u32::from_le_bytes(header[8..12].try_into().map_err(
|_| MemvidError::WalCorruption {
offset: cursor,
reason: "invalid wal length header".into(),
}
})?) as u64;
},
)?));
let checksum = &header[16..48];
if sequence == 0 && length == 0 {
@@ -334,7 +359,13 @@ impl EmbeddedWal {
});
}
let mut payload = vec![0u8; length as usize];
// Safe: length comes from u32::from_le_bytes above, so max is u32::MAX
// which fits in usize on all supported platforms (32-bit and 64-bit)
let length_usize = usize::try_from(length).map_err(|_| MemvidError::WalCorruption {
offset: cursor,
reason: "wal record length too large for platform".into(),
})?;
let mut payload = vec![0u8; length_usize];
file.read_exact(&mut payload)?;
let expected = blake3::hash(&payload);
if expected.as_bytes() != checksum {
@@ -379,8 +410,17 @@ impl EmbeddedWal {
let mut buf = [0u8; ENTRY_HEADER_SIZE];
self.file.seek(SeekFrom::Start(absolute))?;
self.file.read_exact(&mut buf)?;
let seq = u64::from_le_bytes(buf[..8].try_into().unwrap());
let len = u32::from_le_bytes(buf[8..12].try_into().unwrap());
// Safe byte extraction - return early if malformed (debug function)
let seq = buf
.get(..8)
.and_then(|s| <[u8; 8]>::try_from(s).ok())
.map_or(0, u64::from_le_bytes);
let len = buf
.get(8..12)
.and_then(|s| <[u8; 4]>::try_from(s).ok())
.map_or(0, u32::from_le_bytes);
tracing::debug!(
wal.verify_position = pos,
wal.verify_sequence = seq,
@@ -442,13 +482,13 @@ mod tests {
let (file, mut header) = prepare_wal(size);
let mut wal = EmbeddedWal::open(&file, &header).expect("open wal");
wal.append_entry(&vec![0xAA; 32]).expect("append a");
wal.append_entry(&vec![0xBB; 32]).expect("append b");
wal.append_entry(&[0xAA; 32]).expect("append a");
wal.append_entry(&[0xBB; 32]).expect("append b");
wal.record_checkpoint(&mut header).expect("checkpoint");
assert!(wal.pending_records().expect("pending").is_empty());
wal.append_entry(&vec![0xCC; 32]).expect("append c");
wal.append_entry(&[0xCC; 32]).expect("append c");
let records = wal.pending_records().expect("after append");
assert_eq!(records.len(), 1);
assert_eq!(records[0].payload, vec![0xCC; 32]);
+137 -4
View File
@@ -15,6 +15,7 @@ fn lex_config() -> impl bincode::config::Config {
.with_little_endian()
}
#[allow(clippy::cast_possible_truncation)]
const LEX_DECODE_LIMIT: usize = crate::MAX_INDEX_BYTES as usize;
const LEX_SECTION_SOFT_CHARS: usize = 900;
const LEX_SECTION_HARD_CHARS: usize = 1400;
@@ -27,6 +28,7 @@ pub struct LexIndexBuilder {
}
impl LexIndexBuilder {
#[must_use]
pub fn new() -> Self {
Self::default()
}
@@ -149,6 +151,7 @@ impl LexIndex {
Self { documents }
}
#[must_use]
pub fn search(&self, query: &str, limit: usize) -> Vec<LexSearchHit> {
let mut query_tokens = tokenize(query);
query_tokens.retain(|token| !token.is_empty());
@@ -260,6 +263,7 @@ impl LexIndex {
}
occurrences.sort_by_key(|(start, _)| *start);
#[allow(clippy::cast_precision_loss)]
let mut score = occurrences.len() as f32;
if !phrase.is_empty() && section.content_lower.contains(&phrase) {
score += 1000.0;
@@ -277,7 +281,18 @@ impl LexIndex {
}
hits.sort_by(|a, b| b.score.partial_cmp(&a.score).unwrap_or(Ordering::Equal));
hits
// Deduplicate by frame_id, keeping the highest-scoring match for each frame.
// This prevents the same document from appearing multiple times when it has
// multiple sections that match the query.
let mut seen_frames: std::collections::HashSet<FrameId> = std::collections::HashSet::new();
let mut deduped = Vec::with_capacity(hits.len());
for hit in hits {
if seen_frames.insert(hit.frame_id) {
deduped.push(hit);
}
}
deduped
}
}
@@ -402,7 +417,7 @@ fn tokenize(input: &str) -> Vec<String> {
input
.split(|c: char| !is_token_char(c))
.filter_map(|token| {
if token.chars().any(|ch| ch.is_alphanumeric()) {
if token.chars().any(char::is_alphanumeric) {
Some(token.to_lowercase())
} else {
None
@@ -513,7 +528,7 @@ fn push_section(sections: &mut Vec<LexSection>, content: &str, start: usize, end
fn is_soft_boundary(ch: char, next: Option<char>) -> bool {
match ch {
'.' | '!' | '?' => next.map_or(true, |n| n.is_whitespace()),
'.' | '!' | '?' => next.is_none_or(char::is_whitespace),
'\n' => true,
_ => false,
}
@@ -671,7 +686,7 @@ mod tests {
let artifact = builder.finish().expect("finish");
assert_eq!(artifact.doc_count, 2);
assert!(artifact.bytes.len() > 0);
assert!(!artifact.bytes.is_empty());
let index = LexIndex::decode(&artifact.bytes).expect("decode");
let hits = index.search("rust", 10);
@@ -692,4 +707,122 @@ mod tests {
let tokens = tokenize("N&M EXPRESS LLC @ 2024");
assert_eq!(tokens, vec!["n&m", "express", "llc", "2024"]);
}
#[test]
fn compute_matches_deduplicates_by_frame_id() {
// Create a document with content long enough to be split into multiple sections.
// The section soft limit is 900 chars, hard limit is 1400 chars.
// We'll create content > 2000 chars with the search term appearing in each section.
let mut builder = LexIndexBuilder::new();
// Build content with "quantum" appearing in multiple sections
let section1 = "Quantum computing represents a revolutionary approach to computation. \
The fundamental principles of quantum mechanics enable quantum computers to process \
information in ways classical computers cannot. Quantum bits or qubits can exist in \
superposition states, allowing quantum algorithms to explore multiple solutions \
simultaneously. This quantum parallelism offers exponential speedups for certain \
computational problems. Researchers continue to advance quantum hardware and software. \
The field of quantum computing is rapidly evolving with new breakthroughs. \
Major tech companies invest heavily in quantum research and development. \
Quantum error correction remains a significant challenge for practical quantum computers.";
let section2 = "Applications of quantum computing span many domains including cryptography, \
drug discovery, and optimization problems. Quantum cryptography promises unbreakable \
encryption through quantum key distribution protocols. In the pharmaceutical industry, \
quantum simulations could revolutionize how we discover new medicines. Quantum \
algorithms like Shor's algorithm threaten current encryption standards. Financial \
institutions explore quantum computing for portfolio optimization. The quantum \
advantage may soon be demonstrated for practical real-world applications. Quantum \
machine learning combines quantum computing with artificial intelligence techniques. \
The future of quantum computing holds immense promise for scientific discovery.";
let full_content = format!("{} {}", section1, section2);
assert!(
full_content.len() > 1400,
"Content should be long enough to create multiple sections"
);
builder.add_document(
42, // frame_id
"mv2://docs/quantum",
Some("Quantum Computing Overview"),
&full_content,
&HashMap::new(),
);
let artifact = builder.finish().expect("finish should succeed");
let index = LexIndex::decode(&artifact.bytes).expect("decode should succeed");
// Search for "quantum" which appears many times across both sections
let query_tokens = tokenize("quantum");
let matches = index.compute_matches(&query_tokens, None, None);
// Verify: no duplicate frame_ids in results
let frame_ids: Vec<_> = matches.iter().map(|m| m.frame_id).collect();
let unique_frame_ids: std::collections::HashSet<_> = frame_ids.iter().copied().collect();
assert_eq!(
frame_ids.len(),
unique_frame_ids.len(),
"Results should not contain duplicate frame_ids. Found: {:?}",
frame_ids
);
// Should have exactly one result for frame_id 42
assert_eq!(matches.len(), 1, "Should have exactly one match");
assert_eq!(matches[0].frame_id, 42, "Match should be for frame_id 42");
assert!(matches[0].score > 0.0, "Match should have a positive score");
}
#[test]
fn compute_matches_keeps_highest_score_per_frame() {
// Test that when multiple sections match, we keep the highest-scoring one
let mut builder = LexIndexBuilder::new();
// Create content where "target" appears more times in the second section
let section1 = "This is the first section with one target mention. \
It contains various other words to pad the content and make it long enough \
to be split into multiple sections by the chunking algorithm. We need quite \
a bit of text here to ensure the sections are created properly. The content \
continues with more filler text about various topics. Keep writing to reach \
the section boundary. More text follows to ensure we cross the soft limit. \
This should be enough to trigger section creation at the boundary point.";
let section2 = "The second section has target target target multiple times. \
Target appears here repeatedly: target target target target. This section \
should score higher because it has more occurrences of the search term target. \
We mention target again to boost the score further. Target target target. \
The abundance of target keywords makes this section rank higher in relevance.";
let full_content = format!("{} {}", section1, section2);
builder.add_document(
99,
"mv2://docs/multi-section",
Some("Multi-Section Document"),
&full_content,
&HashMap::new(),
);
let artifact = builder.finish().expect("finish");
let index = LexIndex::decode(&artifact.bytes).expect("decode");
let query_tokens = tokenize("target");
let matches = index.compute_matches(&query_tokens, None, None);
// Should have exactly one result (deduplicated)
assert_eq!(
matches.len(),
1,
"Should have exactly one deduplicated match"
);
// The match should have the higher score (from section2 with more "target" occurrences)
// Section1 has 1 occurrence, Section2 has ~10+ occurrences
assert!(
matches[0].score >= 5.0,
"Should keep the highest-scoring match, score was: {}",
matches[0].score
);
}
}
+136 -21
View File
@@ -1,6 +1,75 @@
#![deny(clippy::all, clippy::pedantic)]
#![cfg_attr(not(test), deny(clippy::unwrap_used, clippy::expect_used))]
#![cfg_attr(
test,
allow(
clippy::useless_vec,
clippy::uninlined_format_args,
clippy::cast_possible_truncation,
clippy::float_cmp,
clippy::cast_precision_loss
)
)]
#![allow(clippy::module_name_repetitions)]
//
// Strategic lint exceptions - these are allowed project-wide for pragmatic reasons:
//
// Documentation lints: Many internal/self-documenting functions don't need extensive docs.
// Public APIs should still have proper documentation.
#![allow(clippy::missing_errors_doc)]
#![allow(clippy::missing_panics_doc)]
#![allow(clippy::doc_markdown)]
//
// Cast safety: All casts in this codebase are carefully reviewed and bounded by
// real-world constraints (file sizes, frame counts, etc). Using try_into() everywhere
// would add significant complexity without safety benefits in our use case.
#![allow(clippy::cast_precision_loss)]
#![allow(clippy::cast_possible_wrap)]
#![allow(clippy::cast_sign_loss)]
#![allow(clippy::cast_lossless)]
//
// Style/complexity: Some database-like operations naturally require complex functions.
// Breaking them up would hurt readability.
#![allow(clippy::too_many_lines)]
#![allow(clippy::too_many_arguments)]
#![allow(clippy::items_after_statements)]
#![allow(clippy::similar_names)]
// e.g., frame_id, parent_id, target_id are intentionally similar
//
// Pattern matching: These pedantic lints often suggest changes that reduce clarity.
#![allow(clippy::manual_let_else)]
#![allow(clippy::match_same_arms)]
#![allow(clippy::if_same_then_else)]
#![allow(clippy::collapsible_match)]
//
// Performance/ergonomics trade-offs that are acceptable for this codebase:
#![allow(clippy::needless_pass_by_value)] // Many builders take owned values intentionally
#![allow(clippy::return_self_not_must_use)] // Builder patterns don't need must_use on every method
#![allow(clippy::format_push_string)] // Readability over minor perf difference
#![allow(clippy::assigning_clones)] // clone_from() often less readable
//
// Low-value pedantic lints that add noise:
#![allow(clippy::struct_excessive_bools)] // Config structs naturally have many flags
#![allow(clippy::needless_continue)]
#![allow(clippy::needless_range_loop)]
#![allow(clippy::case_sensitive_file_extension_comparisons)]
#![allow(clippy::default_trait_access)]
#![allow(clippy::field_reassign_with_default)]
#![allow(clippy::unreadable_literal)] // Magic numbers in binary formats are clearer as hex
#![allow(clippy::implicit_hasher)]
#![allow(clippy::manual_clamp)]
#![allow(clippy::len_without_is_empty)] // Many index types don't need is_empty()
#![allow(clippy::large_enum_variant)]
#![allow(clippy::ptr_arg)]
#![allow(clippy::map_unwrap_or)]
#![allow(clippy::incompatible_msrv)]
#![allow(clippy::should_implement_trait)] // Some method names are clearer than trait names
#![allow(clippy::duplicated_attributes)]
//
// Return value wrapping: Many functions use Result for consistency even when they
// currently can't fail, allowing future error conditions to be added without breaking API.
#![allow(clippy::unnecessary_wraps)]
#![allow(clippy::unused_self)] // Some trait impls or future extensibility
/// The memvid-core crate version (matches `Cargo.toml`).
pub const MEMVID_CORE_VERSION: &str = env!("CARGO_PKG_VERSION");
@@ -32,6 +101,12 @@ pub mod types;
pub mod vec;
pub mod vec_pq;
// SIMD-accelerated distance calculations
pub mod simd;
#[cfg(feature = "vec")]
pub mod text_embed;
// Triplet extraction module for automatic SPO extraction during ingestion
pub mod triplet;
@@ -60,6 +135,10 @@ pub mod encryption;
#[cfg(feature = "symspell_cleanup")]
pub mod symspell_cleanup;
// API-based embedding providers (OpenAI, etc.) - requires network
#[cfg(feature = "api_embed")]
pub mod api_embed;
#[cfg(test)]
mod tests_lex_flag;
@@ -105,21 +184,16 @@ pub use models::{
ModelVerifyOptions, verify_model_dir, verify_models,
};
pub use reader::{
DocumentFormat, DocumentReader, PassthroughReader, PdfReader, ReaderDiagnostics, ReaderHint,
ReaderOutput, ReaderRegistry,
DetectedTable, DocumentFormat, DocumentReader, PassthroughReader, PdfReader, ReaderDiagnostics,
ReaderHint, ReaderOutput, ReaderRegistry, XlsxChunkingOptions, XlsxReader,
};
pub use signature::{
parse_ed25519_public_key_base64, verify_model_manifest, verify_ticket_signature,
};
pub use text::{NormalizedText, normalize_text, truncate_at_grapheme_boundary};
#[cfg(feature = "temporal_track")]
pub use types::{
AnchorSource, SearchHitTemporal, SearchHitTemporalAnchor, SearchHitTemporalMention,
TEMPORAL_TRACK_FLAG_HAS_ANCHORS, TEMPORAL_TRACK_FLAG_HAS_MENTIONS, TemporalAnchor,
TemporalCapabilities, TemporalFilter, TemporalMention, TemporalMentionFlags,
TemporalMentionKind, TemporalTrack, TemporalTrackManifest,
};
pub use types::{
ACL_POLICY_VERSION_KEY, ACL_READ_GROUPS_KEY, ACL_READ_PRINCIPALS_KEY, ACL_READ_ROLES_KEY,
ACL_RESOURCE_ID_KEY, ACL_TENANT_ID_KEY, ACL_VISIBILITY_KEY, AclContext, AclEnforcementMode,
AskCitation, AskMode, AskRequest, AskResponse, AskRetriever, AskStats, AudioSegmentMetadata,
AuditOptions, AuditReport, CanonicalEncoding, DOCTOR_PLAN_VERSION, DocAudioMetadata,
DocExifMetadata, DocGpsMetadata, DocMetadata, DoctorActionDetail, DoctorActionKind,
@@ -129,7 +203,7 @@ pub use types::{
EmbeddingIdentity, EmbeddingIdentityCount, EmbeddingIdentitySummary, Frame, FrameId, FrameRole,
FrameStatus, Header, IndexManifests, LexIndexManifest, LexSegmentDescriptor,
MEMVID_EMBEDDING_DIMENSION_KEY, MEMVID_EMBEDDING_MODEL_KEY, MEMVID_EMBEDDING_NORMALIZED_KEY,
MEMVID_EMBEDDING_PROVIDER_KEY, MediaManifest, MemvidHandle, Open, PutOptions,
MEMVID_EMBEDDING_PROVIDER_KEY, MediaManifest, MemvidHandle, Open, PutManyOpts, PutOptions,
PutOptionsBuilder, Sealed, SearchEngineKind, SearchHit, SearchHitMetadata, SearchParams,
SearchRequest, SearchResponse, SegmentCatalog, SegmentCommon, SegmentCompression, SegmentMeta,
SegmentSpan, SourceSpan, Stats, TextChunkManifest, TextChunkRange, Ticket, TicketRef, Tier,
@@ -137,6 +211,13 @@ pub use types::{
Toc, VecEmbedder, VecIndexManifest, VecSegmentDescriptor, VectorCompression, VerificationCheck,
VerificationReport, VerificationStatus,
};
#[cfg(feature = "temporal_track")]
pub use types::{
AnchorSource, SearchHitTemporal, SearchHitTemporalAnchor, SearchHitTemporalMention,
TEMPORAL_TRACK_FLAG_HAS_ANCHORS, TEMPORAL_TRACK_FLAG_HAS_MENTIONS, TemporalAnchor,
TemporalCapabilities, TemporalFilter, TemporalMention, TemporalMentionFlags,
TemporalMentionKind, TemporalTrack, TemporalTrackManifest,
};
// Memory card types for structured memory extraction and storage
pub use types::{
EngineStamp, EnrichmentManifest, EnrichmentRecord, MEMORIES_TRACK_MAGIC,
@@ -192,6 +273,18 @@ pub use vec_pq::{
CompressionStats, ProductQuantizer, QuantizedVecIndex, QuantizedVecIndexArtifact,
QuantizedVecIndexBuilder,
};
// Local text embedding provider - feature-gated
#[cfg(feature = "vec")]
pub use text_embed::{
LocalTextEmbedder, TEXT_EMBED_MODELS, TextEmbedConfig, TextEmbedModelInfo,
default_text_model_info, get_text_model_info,
};
// API-based embedding providers - feature-gated
#[cfg(feature = "api_embed")]
pub use api_embed::{
OPENAI_MODELS, OpenAIConfig, OpenAIEmbedder, OpenAIModelInfo, default_openai_model_info,
get_openai_model_info,
};
// CLIP visual embeddings - types always available for serde compatibility
pub use clip::{
CLIP_MODELS, ClipConfig, ClipDocument, ClipEmbeddingProvider, ClipError, ClipIndex,
@@ -250,6 +343,7 @@ const MAX_FRAME_BYTES: u64 = 256 * 1024 * 1024;
const DEFAULT_SEARCH_TEXT_LIMIT: usize = 32_768;
#[cfg(test)]
#[allow(clippy::non_std_lazy_statics)]
static SERIAL_TEST_MUTEX: Lazy<Mutex<()>> = Lazy::new(|| Mutex::new(()));
#[cfg(test)]
@@ -308,14 +402,17 @@ impl Memvid {
fn flush_tantivy_skip_embed(&mut self) -> Result<()> {
Ok(())
}
#[must_use]
pub fn path(&self) -> &Path {
&self.path
}
#[must_use]
pub fn lock_handle(&self) -> &FileLock {
&self.lock
}
#[must_use]
pub fn is_read_only(&self) -> bool {
self.read_only
}
@@ -391,7 +488,7 @@ pub(crate) fn infer_title_from_uri(uri: &str) -> Option<String> {
return None;
}
let without_scheme = trimmed.splitn(2, "://").nth(1).unwrap_or(trimmed);
let without_scheme = trimmed.split_once("://").map_or(trimmed, |x| x.1);
let without_fragment = without_scheme.split('#').next().unwrap_or(without_scheme);
let without_query = without_fragment
.split('?')
@@ -406,13 +503,13 @@ pub(crate) fn infer_title_from_uri(uri: &str) -> Option<String> {
return None;
}
let stem = segment.rsplitn(2, '.').nth(1).unwrap_or(segment).trim();
let stem = segment.rsplit_once('.').map_or(segment, |x| x.0).trim();
if stem.is_empty() {
return None;
}
let words: Vec<String> = stem
.split(|c: char| c == '-' || c == '_' || c == ' ')
.split(['-', '_', ' '])
.filter(|part| !part.is_empty())
.map(|part| {
let mut chars = part.chars();
@@ -423,7 +520,7 @@ pub(crate) fn infer_title_from_uri(uri: &str) -> Option<String> {
if rest.is_empty() {
first.to_string()
} else {
format!("{}{}", first, rest)
format!("{first}{rest}")
}
}
None => String::new(),
@@ -458,7 +555,7 @@ fn image_preview_from_metadata(meta: &DocMetadata) -> Option<String> {
let mut segments: Vec<String> = Vec::new();
if let (Some(w), Some(h)) = (meta.width, meta.height) {
segments.push(format!("{}×{} px", w, h));
segments.push(format!("{w}×{h} px"));
}
if let Some(exif) = meta.exif.as_ref() {
if let Some(model) = exif
@@ -598,6 +695,8 @@ mod tests {
as_of_frame: None,
as_of_ts: None,
no_sketch: false,
acl_context: None,
acl_enforcement_mode: crate::types::AclEnforcementMode::Audit,
};
let response = mem.search(request).expect("search");
assert_eq!(response.hits.len(), 1);
@@ -617,6 +716,8 @@ mod tests {
as_of_frame: None,
as_of_ts: None,
no_sketch: false,
acl_context: None,
acl_enforcement_mode: crate::types::AclEnforcementMode::Audit,
};
let response = reopened.search(request).expect("search reopened");
assert_eq!(response.hits.len(), 1);
@@ -688,6 +789,8 @@ mod tests {
as_of_frame: None,
as_of_ts: None,
no_sketch: false,
acl_context: None,
acl_enforcement_mode: crate::types::AclEnforcementMode::Audit,
})
.expect("search");
@@ -749,6 +852,8 @@ mod tests {
as_of_frame: None,
as_of_ts: None,
no_sketch: false,
acl_context: None,
acl_enforcement_mode: crate::types::AclEnforcementMode::Audit,
})
.expect("search");
@@ -833,6 +938,8 @@ mod tests {
as_of_frame: None,
as_of_ts: None,
no_sketch: false,
acl_context: None,
acl_enforcement_mode: crate::types::AclEnforcementMode::Audit,
})
.expect("uri search");
assert_eq!(uri_response.engine, SearchEngineKind::Tantivy);
@@ -856,6 +963,8 @@ mod tests {
as_of_frame: None,
as_of_ts: None,
no_sketch: false,
acl_context: None,
acl_enforcement_mode: crate::types::AclEnforcementMode::Audit,
})
.expect("scope search");
assert_eq!(scope_response.engine, SearchEngineKind::Tantivy);
@@ -909,6 +1018,8 @@ mod tests {
as_of_frame: None,
as_of_ts: None,
no_sketch: false,
acl_context: None,
acl_enforcement_mode: crate::types::AclEnforcementMode::Audit,
})
.expect("page one");
assert_eq!(first_page.engine, SearchEngineKind::Tantivy);
@@ -932,6 +1043,8 @@ mod tests {
as_of_frame: None,
as_of_ts: None,
no_sketch: false,
acl_context: None,
acl_enforcement_mode: crate::types::AclEnforcementMode::Audit,
})
.expect("page two");
assert_eq!(second_page.engine, SearchEngineKind::Tantivy);
@@ -973,6 +1086,8 @@ mod tests {
as_of_frame: None,
as_of_ts: None,
no_sketch: false,
acl_context: None,
acl_enforcement_mode: crate::types::AclEnforcementMode::Audit,
})
.expect("search with tantivy");
@@ -1055,7 +1170,7 @@ mod tests {
mem.put_bytes(b"repair").expect("put");
mem.commit().expect("commit");
// Explicitly rebuild indexes to create time_index (new implementation requires this)
mem.rebuild_indexes(&[]).expect("rebuild");
mem.rebuild_indexes(&[], &[]).expect("rebuild");
mem.commit().expect("commit after rebuild");
println!(
"test: post-commit header footer_offset={}",
@@ -1085,7 +1200,7 @@ mod tests {
.expect("open file");
file.seek(SeekFrom::Start(manifest.bytes_offset))
.expect("seek");
let zeros = vec![0u8; manifest.bytes_length as usize];
let zeros = vec![0u8; usize::try_from(manifest.bytes_length).unwrap_or(0)];
file.write_all(&zeros).expect("corrupt time index");
file.flush().expect("flush");
file.sync_all().expect("sync");
@@ -1103,7 +1218,7 @@ mod tests {
assert_eq!(report.overall_status, VerificationStatus::Failed);
}
Err(e) => {
println!("test: verify failed with error (expected): {}", e);
println!("test: verify failed with error (expected): {e}");
}
}
@@ -1255,6 +1370,7 @@ mod tests {
}
#[test]
#[allow(deprecated)]
fn ticket_sequence_enforced() {
run_serial_test(|| {
let dir = tempdir().expect("tmp");
@@ -1272,6 +1388,7 @@ mod tests {
}
#[test]
#[allow(deprecated)]
fn capacity_limit_enforced() {
run_serial_test(|| {
let dir = tempdir().expect("tmp");
@@ -1285,9 +1402,7 @@ mod tests {
mem.put_bytes(&vec![0xFF; 32]).expect("first put");
mem.commit().expect("commit");
let err = mem
.put_bytes(&vec![0xFF; 40])
.expect_err("capacity exceeded");
let err = mem.put_bytes(&[0xFF; 40]).expect_err("capacity exceeded");
assert!(matches!(err, MemvidError::CapacityExceeded { .. }));
});
}
+2
View File
@@ -81,6 +81,7 @@ impl FileLock {
Ok(self.file.try_clone()?)
}
#[must_use]
pub fn mode(&self) -> LockMode {
self.mode
}
@@ -170,6 +171,7 @@ mod tests {
use tempfile::NamedTempFile;
#[test]
#[cfg(not(target_os = "windows"))] // Windows has different file locking semantics
fn acquiring_lock_blocks_second_writer() {
let temp = NamedTempFile::new().expect("temp file");
let path = temp.path();
+17 -6
View File
@@ -35,7 +35,7 @@ pub struct LockOptions<'a> {
pub force_stale: bool,
}
impl<'a> Default for LockOptions<'a> {
impl Default for LockOptions<'_> {
fn default() -> Self {
Self {
timeout: Duration::from_millis(DEFAULT_TIMEOUT_MS),
@@ -48,26 +48,31 @@ impl<'a> Default for LockOptions<'a> {
}
impl<'a> LockOptions<'a> {
#[must_use]
pub fn timeout_ms(mut self, timeout_ms: u64) -> Self {
self.timeout = Duration::from_millis(timeout_ms);
self
}
#[must_use]
pub fn heartbeat_ms(mut self, heartbeat_ms: u64) -> Self {
self.heartbeat = Duration::from_millis(heartbeat_ms);
self
}
#[must_use]
pub fn stale_grace_ms(mut self, stale_grace_ms: u64) -> Self {
self.stale_grace = Duration::from_millis(stale_grace_ms);
self
}
#[must_use]
pub fn command(mut self, command: &'a str) -> Self {
self.command = Some(command);
self
}
#[must_use]
pub fn force_stale(mut self, force: bool) -> Self {
self.force_stale = force;
self
@@ -95,10 +100,12 @@ impl LockfileGuard {
Ok(())
}
#[must_use]
pub fn file_id(&self) -> &FileId {
&self.file_id
}
#[must_use]
pub fn owner_hint(&self) -> LockOwnerHint {
self.record.to_owner_hint()
}
@@ -116,8 +123,7 @@ pub fn acquire(path: &Path, options: LockOptions<'_>) -> Result<LockfileGuard> {
let file_id = registry::compute_file_id(path)?;
let command = options
.command
.map(std::borrow::ToOwned::to_owned)
.unwrap_or_else(default_command);
.map_or_else(default_command, std::borrow::ToOwned::to_owned);
let heartbeat_ms = options
.heartbeat
.as_millis()
@@ -149,8 +155,7 @@ pub fn acquire(path: &Path, options: LockOptions<'_>) -> Result<LockfileGuard> {
let existing = registry::read_record(&file_id)?;
let stale = existing
.as_ref()
.map(|rec| registry::is_stale(rec, options.stale_grace))
.unwrap_or(true);
.is_none_or(|rec| registry::is_stale(rec, options.stale_grace));
if options.force_stale && stale {
let _ = registry::remove_record(&file_id);
@@ -172,7 +177,13 @@ pub fn acquire(path: &Path, options: LockOptions<'_>) -> Result<LockfileGuard> {
)
})
.unwrap_or_else(|| "memory locked by another process".to_string());
return Err(LockedError::new(path.to_path_buf(), message, hint, stale).into());
return Err(Box::new(LockedError::new(
path.to_path_buf(),
message,
hint,
stale,
))
.into());
}
let remaining = options
+339
View File
@@ -0,0 +1,339 @@
use std::collections::{BTreeMap, HashSet};
use crate::memvid::lifecycle::Memvid;
use crate::types::{
ACL_READ_GROUPS_KEY, ACL_READ_PRINCIPALS_KEY, ACL_READ_ROLES_KEY, ACL_TENANT_ID_KEY,
ACL_VISIBILITY_KEY, AclContext, AclEnforcementMode, SearchHit,
};
use crate::{MemvidError, Result};
#[derive(Debug, Clone, Default)]
pub(crate) struct AclFilterStats {
pub allowed: usize,
pub denied: usize,
pub cross_tenant_denied: usize,
pub missing_metadata: usize,
}
impl AclFilterStats {
fn record(&mut self, decision: AclDecision) {
if decision.allowed {
self.allowed += 1;
return;
}
self.denied += 1;
if decision.cross_tenant_denied {
self.cross_tenant_denied += 1;
}
if decision.missing_metadata_denied {
self.missing_metadata += 1;
}
}
}
#[derive(Debug, Clone, Eq, PartialEq)]
struct NormalizedAclContext {
tenant_id: String,
subject_id: Option<String>,
roles: HashSet<String>,
group_ids: HashSet<String>,
}
#[derive(Debug, Clone)]
struct ParsedFrameAcl {
tenant_id: String,
visibility: FrameVisibility,
roles: HashSet<String>,
groups: HashSet<String>,
principals: HashSet<String>,
}
#[derive(Debug, Clone, Copy, Eq, PartialEq)]
enum FrameVisibility {
Public,
Restricted,
}
#[derive(Debug, Clone, Copy, Default)]
struct AclDecision {
allowed: bool,
cross_tenant_denied: bool,
missing_metadata_denied: bool,
}
impl AclDecision {
fn allow() -> Self {
Self {
allowed: true,
..Self::default()
}
}
fn deny_cross_tenant() -> Self {
Self {
allowed: false,
cross_tenant_denied: true,
missing_metadata_denied: false,
}
}
fn deny_missing_metadata() -> Self {
Self {
allowed: false,
cross_tenant_denied: false,
missing_metadata_denied: true,
}
}
fn deny_restricted() -> Self {
Self {
allowed: false,
..Self::default()
}
}
}
impl Memvid {
pub(crate) fn apply_acl_to_search_hits(
&self,
hits: &mut Vec<SearchHit>,
acl_context: Option<&AclContext>,
acl_enforcement_mode: AclEnforcementMode,
) -> Result<AclFilterStats> {
let normalized_context = match acl_enforcement_mode {
AclEnforcementMode::Audit => normalize_acl_context(acl_context),
AclEnforcementMode::Enforce => Some(validate_enforce_acl_context(acl_context)?),
};
let mut stats = AclFilterStats::default();
if normalized_context.is_none() {
stats.allowed = hits.len();
return Ok(stats);
}
let mut filtered_hits = Vec::with_capacity(hits.len());
for hit in &*hits {
let decision = match self.frame_by_id(hit.frame_id) {
Ok(frame) => {
evaluate_acl_metadata(&frame.extra_metadata, normalized_context.as_ref())
}
Err(_) => AclDecision::deny_missing_metadata(),
};
stats.record(decision);
if decision.allowed || acl_enforcement_mode == AclEnforcementMode::Audit {
filtered_hits.push(hit.clone());
}
}
if acl_enforcement_mode == AclEnforcementMode::Enforce {
for (index, hit) in filtered_hits.iter_mut().enumerate() {
hit.rank = index + 1;
}
*hits = filtered_hits;
}
Ok(stats)
}
}
fn validate_enforce_acl_context(context: Option<&AclContext>) -> Result<NormalizedAclContext> {
let Some(context) = context else {
return Err(MemvidError::InvalidQuery {
reason: "acl_context is required when acl_enforcement_mode is 'enforce'".to_string(),
});
};
let Some(normalized) = normalize_acl_context(Some(context)) else {
return Err(MemvidError::InvalidQuery {
reason: "acl_context.tenant_id is required when acl_enforcement_mode is 'enforce'"
.to_string(),
});
};
Ok(normalized)
}
fn normalize_acl_context(context: Option<&AclContext>) -> Option<NormalizedAclContext> {
let context = context?;
let tenant_id = normalize_scalar(context.tenant_id.as_deref())?;
let subject_id = context
.subject_id
.as_deref()
.and_then(|value| normalize_scalar(Some(value)));
let roles = context
.roles
.iter()
.filter_map(|role| normalize_scalar(Some(role.as_str())))
.collect();
let group_ids = context
.group_ids
.iter()
.filter_map(|group| normalize_scalar(Some(group.as_str())))
.collect();
Some(NormalizedAclContext {
tenant_id,
subject_id,
roles,
group_ids,
})
}
fn evaluate_acl_metadata(
metadata: &BTreeMap<String, String>,
context: Option<&NormalizedAclContext>,
) -> AclDecision {
let Some(context) = context else {
return AclDecision::allow();
};
let parsed = match parse_acl_metadata(metadata) {
Ok(parsed) => parsed,
Err(()) => return AclDecision::deny_missing_metadata(),
};
if parsed.tenant_id != context.tenant_id {
return AclDecision::deny_cross_tenant();
}
if parsed.visibility == FrameVisibility::Public {
return AclDecision::allow();
}
let principal_allowed = context
.subject_id
.as_ref()
.is_some_and(|subject| parsed.principals.contains(subject));
let role_allowed = context.roles.iter().any(|role| parsed.roles.contains(role));
let group_allowed = context
.group_ids
.iter()
.any(|group| parsed.groups.contains(group));
if principal_allowed || role_allowed || group_allowed {
AclDecision::allow()
} else {
AclDecision::deny_restricted()
}
}
fn parse_acl_metadata(
metadata: &BTreeMap<String, String>,
) -> std::result::Result<ParsedFrameAcl, ()> {
let tenant_id =
normalize_scalar(metadata.get(ACL_TENANT_ID_KEY).map(String::as_str)).ok_or(())?;
let visibility_raw =
normalize_scalar(metadata.get(ACL_VISIBILITY_KEY).map(String::as_str)).ok_or(())?;
let visibility = match visibility_raw.as_str() {
"public" => FrameVisibility::Public,
"restricted" => FrameVisibility::Restricted,
_ => return Err(()),
};
let roles = parse_acl_list(metadata, ACL_READ_ROLES_KEY)?;
let groups = parse_acl_list(metadata, ACL_READ_GROUPS_KEY)?;
let principals = parse_acl_list(metadata, ACL_READ_PRINCIPALS_KEY)?;
Ok(ParsedFrameAcl {
tenant_id,
visibility,
roles,
groups,
principals,
})
}
fn parse_acl_list(
metadata: &BTreeMap<String, String>,
key: &str,
) -> std::result::Result<HashSet<String>, ()> {
let Some(raw) = metadata.get(key) else {
return Ok(HashSet::new());
};
let values: Vec<String> = serde_json::from_str(raw).map_err(|_| ())?;
let mut parsed = HashSet::with_capacity(values.len());
for value in values {
let normalized = normalize_scalar(Some(value.as_str())).ok_or(())?;
parsed.insert(normalized);
}
Ok(parsed)
}
fn normalize_scalar(value: Option<&str>) -> Option<String> {
let value = value?;
let trimmed = value.trim();
if trimmed.is_empty() {
None
} else {
// Accept legacy/stringified metadata values emitted by some bindings,
// e.g. acl_visibility stored as "\"restricted\"" instead of "restricted".
let unwrapped = match serde_json::from_str::<String>(trimmed) {
Ok(parsed) => parsed.trim().to_string(),
Err(_) => trimmed.to_string(),
};
if unwrapped.is_empty() {
None
} else {
Some(unwrapped.to_ascii_lowercase())
}
}
}
#[cfg(test)]
mod tests {
use super::*;
fn restricted_metadata() -> BTreeMap<String, String> {
BTreeMap::from([
(ACL_TENANT_ID_KEY.to_string(), "tenant-a".to_string()),
(ACL_VISIBILITY_KEY.to_string(), "restricted".to_string()),
(
ACL_READ_ROLES_KEY.to_string(),
"[\"admin\",\"analyst\"]".to_string(),
),
(ACL_READ_GROUPS_KEY.to_string(), "[\"eng\"]".to_string()),
(
ACL_READ_PRINCIPALS_KEY.to_string(),
"[\"user-123\"]".to_string(),
),
])
}
fn context(tenant: &str) -> NormalizedAclContext {
NormalizedAclContext {
tenant_id: tenant.to_string(),
subject_id: Some("user-123".to_string()),
roles: HashSet::from(["viewer".to_string()]),
group_ids: HashSet::from(["eng".to_string()]),
}
}
#[test]
fn parse_acl_metadata_rejects_invalid_list_encoding() {
let mut metadata = restricted_metadata();
metadata.insert(ACL_READ_GROUPS_KEY.to_string(), "eng,ops".to_string());
assert!(parse_acl_metadata(&metadata).is_err());
}
#[test]
fn evaluate_acl_denies_cross_tenant() {
let metadata = restricted_metadata();
let ctx = context("tenant-b");
let decision = evaluate_acl_metadata(&metadata, Some(&ctx));
assert!(!decision.allowed);
assert!(decision.cross_tenant_denied);
}
#[test]
fn evaluate_acl_allows_restricted_group_match() {
let metadata = restricted_metadata();
let ctx = context("tenant-a");
let decision = evaluate_acl_metadata(&metadata, Some(&ctx));
assert!(decision.allowed);
}
#[test]
fn evaluate_acl_denies_missing_metadata() {
let metadata = BTreeMap::new();
let ctx = context("tenant-a");
let decision = evaluate_acl_metadata(&metadata, Some(&ctx));
assert!(!decision.allowed);
assert!(decision.missing_metadata_denied);
}
}
+53 -28
View File
@@ -1,3 +1,5 @@
// Safe unwrap: float comparisons with fallback ordering.
#![allow(clippy::unwrap_used)]
use std::borrow::Cow;
use std::collections::{BTreeSet, HashMap, HashSet};
use std::num::NonZeroU64;
@@ -30,7 +32,7 @@ impl Memvid {
let lexical_query = sanitize_question_for_lexical(&request.question);
let primary_tokens: Vec<String> = lexical_query
.split_whitespace()
.map(|token| token.to_ascii_lowercase())
.map(str::to_ascii_lowercase)
.collect();
// Detect aggregation questions that need multi-session retrieval
@@ -103,6 +105,8 @@ impl Memvid {
// Disable sketch pre-filter for ask queries - accuracy is more important than speed
// SimHash can filter out semantically relevant documents that use different wording
no_sketch: true,
acl_context: request.acl_context.clone(),
acl_enforcement_mode: request.acl_enforcement_mode,
};
// Pre-compute the query embedding once so we can reuse it for vector recall and semantic re-rank
@@ -364,6 +368,15 @@ impl Memvid {
// This ensures user corrections override all other ranking signals
promote_corrections(self, &mut retrieval.hits)?;
self.apply_acl_to_search_hits(
&mut retrieval.hits,
request.acl_context.as_ref(),
request.acl_enforcement_mode,
)?;
if request.acl_enforcement_mode == crate::types::AclEnforcementMode::Enforce {
retrieval.total_hits = retrieval.hits.len();
}
retrieval.context = build_context(&retrieval.hits);
let (answer, citations, synthesis_ms) = if request.context_only {
@@ -500,12 +513,17 @@ impl Memvid {
.map(|manifest| manifest.dimension)
.filter(|dim| *dim > 0)
.or_else(|| {
self.vec_index
.as_ref()
.and_then(|index| index.entries().next().map(|(_, emb)| emb.len() as u32))
self.vec_index.as_ref().and_then(|index| {
index
.entries()
.next()
.map(|(_, emb)| u32::try_from(emb.len()).unwrap_or(0))
})
})
.unwrap_or(0);
if stored_dimension > 0 && query_embedding.len() as u32 != stored_dimension {
if stored_dimension > 0
&& u32::try_from(query_embedding.len()).unwrap_or(u32::MAX) != stored_dimension
{
return Err(MemvidError::VecDimensionMismatch {
expected: stored_dimension,
actual: query_embedding.len(),
@@ -516,7 +534,7 @@ impl Memvid {
for hit in hits.iter() {
if let Some(embedding) = self.frame_embedding(hit.frame_id)? {
if expected_dimension == 0 || embedding.len() == expected_dimension {
let score = cosine_similarity(&query_embedding, &embedding);
let score = cosine_similarity(query_embedding, &embedding);
semantic_scores.insert(hit.frame_id, score);
}
}
@@ -530,7 +548,7 @@ impl Memvid {
Ok(true)
}
/// Build a fallback SearchResponse from timeline entries when search returns no hits.
/// Build a fallback `SearchResponse` from timeline entries when search returns no hits.
/// This gives the LLM some context to work with for general questions about the document.
/// For comprehensive coverage, includes child frames (e.g., document pages) as well.
fn build_timeline_fallback_response(
@@ -558,13 +576,14 @@ impl Memvid {
snippet_chars: request.snippet_chars,
cursor: search_request.cursor.clone(),
},
stale_index_skips: 0,
});
}
// Collect all frame IDs including child frames for comprehensive coverage
// This is critical for analytical questions that need full document context
let mut all_frame_ids: Vec<(u64, Option<String>)> = Vec::new();
for entry in entries.iter() {
for entry in &entries {
// Add parent frame
all_frame_ids.push((entry.frame_id, entry.uri.clone()));
// Add all child frames (e.g., document pages)
@@ -592,7 +611,7 @@ impl Memvid {
.uri
.clone()
.or_else(|| parent_uri.clone())
.unwrap_or_else(|| format!("mv2://frame/{}", frame_id));
.unwrap_or_else(|| format!("mv2://frame/{frame_id}"));
(content, uri)
}
Err(_) => continue, // Skip frames we can't read
@@ -641,6 +660,7 @@ impl Memvid {
snippet_chars: request.snippet_chars,
cursor: search_request.cursor.clone(),
},
stale_index_skips: 0,
})
}
}
@@ -716,8 +736,7 @@ fn reorder_hits_with_semantic_scores(
let lexical_rrf = 1.0 / (RRF_K + lexical_rank as f32);
let semantic_rrf = semantic_rank
.get(&hit.frame_id)
.map(|rank| 1.0 / (RRF_K + *rank as f32))
.unwrap_or(0.0);
.map_or(0.0, |rank| 1.0 / (RRF_K + *rank as f32));
semantic_score + lexical_rrf + semantic_rrf
}
AskMode::Lex => 1.0 / (RRF_K + lexical_rank as f32),
@@ -818,7 +837,7 @@ fn lexical_fallback_query(question: &str) -> Option<String> {
let sanitized_tokens: Vec<String> = sanitized_full
.split_whitespace()
.map(|token| token.to_string())
.map(std::string::ToString::to_string)
.collect();
let mut candidates: Vec<String> = question
@@ -867,7 +886,7 @@ fn is_stopword(token: &str) -> bool {
"was", "we", "were", "what", "when", "where", "which", "who", "whom", "why", "with", "you",
"your", "yours",
];
STOPWORDS.iter().any(|stop| *stop == token)
STOPWORDS.contains(&token)
}
fn sanitize_question_for_lexical(question: &str) -> String {
@@ -944,7 +963,7 @@ fn build_expanded_queries(tokens: &[String]) -> Vec<String> {
let key_nouns: Vec<&str> = tokens
.iter()
.filter(|t| !is_stopword(t) && t.len() > 3)
.map(|t| t.as_str())
.map(std::string::String::as_str)
.collect();
if key_nouns.is_empty() {
@@ -954,7 +973,7 @@ fn build_expanded_queries(tokens: &[String]) -> Vec<String> {
// For each key noun, try singular/plural and possessive forms
for noun in &key_nouns {
// Try the base form
variants.push(noun.to_string());
variants.push((*noun).to_string());
// Try singular/plural variants
if noun.ends_with('s') && noun.len() > 4 {
@@ -963,16 +982,16 @@ fn build_expanded_queries(tokens: &[String]) -> Vec<String> {
variants.push(singular.to_string());
} else if !noun.ends_with('s') {
// "wedding" -> "weddings"
variants.push(format!("{}s", noun));
variants.push(format!("{noun}s"));
}
}
// Create OR queries from the variants
if !variants.is_empty() {
if variants.is_empty() {
Vec::new()
} else {
let or_query = variants.join(" OR ");
vec![or_query]
} else {
Vec::new()
}
}
@@ -1094,7 +1113,7 @@ fn is_recency_question(question: &str) -> bool {
"up to date",
];
for pattern in multi_word_patterns.iter() {
for pattern in &multi_word_patterns {
if lower.contains(pattern) {
return true;
}
@@ -1113,8 +1132,8 @@ fn is_recency_question(question: &str) -> bool {
// Split into words and check for exact matches
let words: Vec<&str> = lower.split(|c: char| !c.is_alphanumeric()).collect();
for pattern in single_word_patterns.iter() {
if words.iter().any(|w| *w == *pattern) {
for pattern in &single_word_patterns {
if words.contains(pattern) {
return true;
}
}
@@ -1178,7 +1197,7 @@ fn build_analytical_query(tokens: &[String]) -> String {
// Keep only content-bearing terms
let content_terms: Vec<&str> = tokens
.iter()
.map(|t| t.as_str())
.map(std::string::String::as_str)
.filter(|t| !analytical_stopwords.contains(*t) && t.len() > 2)
.collect();
@@ -1231,7 +1250,7 @@ fn is_analytical_question(question: &str) -> bool {
"any differences",
];
for pattern in analytical_patterns.iter() {
for pattern in &analytical_patterns {
if lower.contains(pattern) {
return true;
}
@@ -1326,12 +1345,14 @@ fn vector_hits(
// Use adaptive retrieval if configured
if let Some(ref adaptive_config) = request.adaptive {
if adaptive_config.enabled {
let result = memvid.search_adaptive(
let result = memvid.search_adaptive_acl(
&request.question,
query_embedding,
adaptive_config.clone(),
request.snippet_chars,
request.scope.as_deref(),
request.acl_context.as_ref(),
request.acl_enforcement_mode,
)?;
tracing::debug!(
"adaptive retrieval: {} -> {} results ({})",
@@ -1343,12 +1364,14 @@ fn vector_hits(
}
}
let vec_response = memvid.vec_search_with_embedding(
let vec_response = memvid.vec_search_with_embedding_acl(
&request.question,
query_embedding,
limit,
request.snippet_chars,
request.scope.as_deref(),
request.acl_context.as_ref(),
request.acl_enforcement_mode,
)?;
Ok(vec_response.hits)
@@ -1440,7 +1463,8 @@ fn promote_corrections(memvid: &mut Memvid, hits: &mut Vec<SearchHit>) -> Result
// Sort corrections by timestamp DESC (newest first), then by boost DESC
corrections.sort_by(|a, b| {
b.1.cmp(&a.1) // timestamp descending (newest first)
.then_with(|| b.2.partial_cmp(&a.2).unwrap_or(std::cmp::Ordering::Equal)) // boost descending
.then_with(|| b.2.partial_cmp(&a.2).unwrap_or(std::cmp::Ordering::Equal))
// boost descending
});
tracing::debug!(
@@ -1470,8 +1494,9 @@ fn promote_corrections(memvid: &mut Memvid, hits: &mut Vec<SearchHit>) -> Result
}
/// Promote earliest/latest hits into the visible context so update/recency questions see both ends.
/// Uses content_dates for temporal ordering (dates extracted from document content),
/// Uses `content_dates` for temporal ordering (dates extracted from document content),
/// falling back to frame.timestamp (ingestion time) if no content dates are available.
#[cfg(feature = "lex")]
fn promote_temporal_extremes(
memvid: &mut Memvid,
hits: &mut Vec<SearchHit>,
+2
View File
@@ -71,6 +71,8 @@ impl Memvid {
as_of_frame: None,
as_of_ts: None,
adaptive: None,
acl_context: None,
acl_enforcement_mode: crate::types::AclEnforcementMode::Audit,
};
let response = self.ask(request, embedder)?;
+1
View File
@@ -280,6 +280,7 @@ impl Memvid {
bytes_length: 0,
checksum: empty_checksum,
compression_mode: self.vec_compression.clone(),
model: None,
});
}
if let Some(manifest) = self.toc.indexes.vec.as_mut() {
+5 -5
View File
@@ -85,7 +85,7 @@ fn plan_structural_chunks(
Some(DocumentChunkPlan { manifest, chunks })
}
/// Build TextChunkManifest from structural chunks.
/// Build `TextChunkManifest` from structural chunks.
fn build_manifest_from_structural(
chunks: &[crate::structure::StructuredChunk],
text: &str,
@@ -297,7 +297,7 @@ mod tests {
#[test]
fn structural_chunking_keeps_small_table_whole() {
let text = r#"# Small Report
let text = r"# Small Report
Introduction paragraph.
@@ -307,7 +307,7 @@ Introduction paragraph.
| Orange | $2 |
Conclusion.
"#
"
.repeat(50); // Repeat to meet minimum size
let plan = plan_document_chunks(text.as_bytes()).expect("chunk plan");
@@ -327,7 +327,7 @@ Conclusion.
#[test]
fn structural_chunking_detects_code_blocks() {
let text = r#"# Code Example
let text = r"# Code Example
Here is some code:
@@ -347,7 +347,7 @@ class DataProcessor:
self.data.append(item)
```
More explanation here. "#
More explanation here. "
.repeat(20);
let plan = plan_document_chunks(text.as_bytes()).expect("chunk plan");
+146 -98
View File
@@ -1,3 +1,6 @@
// Safe unwrap/expect: Option takes with immediate value replacement.
#![allow(clippy::unwrap_used, clippy::expect_used)]
use std::cell::Cell;
use std::cmp::min;
use std::fs::OpenOptions;
use std::io::{Read, Seek, SeekFrom, Write};
@@ -22,6 +25,30 @@ use crate::types::{Header, Toc};
use crate::lex::LexIndex;
use crate::vec::VecIndex;
// Thread-local flag to control doctor debug logging
thread_local! {
static DOCTOR_QUIET: Cell<bool> = const { Cell::new(false) };
}
/// Set quiet mode for doctor logging (suppresses debug output when true).
fn set_doctor_quiet(quiet: bool) {
DOCTOR_QUIET.with(|q| q.set(quiet));
}
/// Check if doctor logging is suppressed.
fn is_doctor_quiet() -> bool {
DOCTOR_QUIET.with(std::cell::Cell::get)
}
/// Conditionally print doctor debug messages based on quiet flag.
macro_rules! doctor_log {
($($arg:tt)*) => {
if !is_doctor_quiet() {
println!($($arg)*);
}
};
}
#[derive(Default)]
struct IndexProbe {
needs_time: bool,
@@ -47,9 +74,12 @@ struct PlanProbe {
}
pub(crate) fn doctor_plan(path: &Path, options: DoctorOptions) -> Result<DoctorPlan> {
println!(
doctor_log!(
"doctor: planning for {:?} (options: rebuild_time={}, rebuild_lex={}, rebuild_vec={})",
path, options.rebuild_time_index, options.rebuild_lex_index, options.rebuild_vec_index,
path,
options.rebuild_time_index,
options.rebuild_lex_index,
options.rebuild_vec_index,
);
ensure_single_file(path)?;
let planner = DoctorPlanner::new(path.to_path_buf(), options);
@@ -60,21 +90,23 @@ pub(crate) fn doctor_plan(path: &Path, options: DoctorOptions) -> Result<DoctorP
fn try_recover_from_wal_corruption(path: &Path) -> Result<Memvid> {
use fs2::FileExt;
println!("doctor: opening file for WAL recovery");
doctor_log!("doctor: opening file for WAL recovery");
let mut file = OpenOptions::new().read(true).write(true).open(path)?;
// Acquire exclusive lock
file.lock_exclusive()?;
println!("doctor: reading header");
doctor_log!("doctor: reading header");
let mut header = HeaderCodec::read(&mut file)?;
println!(
doctor_log!(
"doctor: zeroing out corrupted WAL region (offset: {}, size: {})",
header.wal_offset, header.wal_size
header.wal_offset,
header.wal_size
);
// Zero out the entire WAL region to create a clean slate
#[allow(clippy::cast_possible_truncation)]
let wal_size = header.wal_size as usize;
let zeros = vec![0u8; min(1024 * 1024, wal_size)]; // Write in 1MB chunks
let mut written = 0;
@@ -90,13 +122,13 @@ fn try_recover_from_wal_corruption(path: &Path) -> Result<Memvid> {
header.wal_checkpoint_pos = 0;
header.wal_sequence = 0;
println!("doctor: writing repaired header");
doctor_log!("doctor: writing repaired header");
HeaderCodec::write(&mut file, &header)?;
// Flush all changes to disk
file.sync_all()?;
println!("doctor: WAL rebuilt, attempting to open memory");
doctor_log!("doctor: WAL rebuilt, attempting to open memory");
// Now try to open the file normally - WAL should be clean
// We need to release the lock first, then reopen
@@ -128,9 +160,10 @@ pub(crate) fn doctor_apply(path: &Path, plan: DoctorPlan) -> Result<DoctorReport
}
pub(crate) fn doctor_run(path: &Path, options: DoctorOptions) -> Result<DoctorReport> {
println!("doctor: doctor_run start");
set_doctor_quiet(options.quiet);
doctor_log!("doctor: doctor_run start");
let plan = doctor_plan(path, options.clone())?;
println!("doctor: plan ready");
doctor_log!("doctor: plan ready");
doctor_apply(path, plan)
}
@@ -145,10 +178,10 @@ impl DoctorPlanner {
}
fn compute(mut self) -> Result<DoctorPlan> {
println!("doctor: planner.compute start");
doctor_log!("doctor: planner.compute start");
let start = std::time::Instant::now();
let mut probe = self.probe()?;
println!(
doctor_log!(
"doctor: probe complete in {:?} (wal_pending={}, findings={})",
start.elapsed(),
probe.wal_pending,
@@ -169,7 +202,7 @@ impl DoctorPlanner {
action: DoctorActionKind::HealHeaderPointer,
required: true,
reasons: vec![DoctorFindingCode::HeaderFooterOffsetMismatch],
note: Some(format!("heal footer offset to {}", offset)),
note: Some(format!("heal footer offset to {offset}")),
detail: Some(DoctorActionDetail::HeaderPointer {
target_footer_offset: offset,
}),
@@ -327,7 +360,7 @@ impl DoctorPlanner {
}
fn probe(&mut self) -> Result<PlanProbe> {
println!("doctor: probe start");
doctor_log!("doctor: probe start");
let mut probe = PlanProbe {
header: None,
toc: None,
@@ -343,9 +376,9 @@ impl DoctorPlanner {
let mut file = OpenOptions::new().read(true).write(true).open(&self.path)?;
probe.file_len = file.metadata()?.len();
println!("doctor: probe file len {}", probe.file_len);
doctor_log!("doctor: probe file len {}", probe.file_len);
println!("doctor: reading header");
doctor_log!("doctor: reading header");
match HeaderCodec::read(&mut file) {
Ok(header) => probe.header = Some(header),
Err(err) => {
@@ -360,17 +393,19 @@ impl DoctorPlanner {
let Some(header) = probe.header.as_ref() else {
return Ok(probe);
};
println!(
doctor_log!(
"doctor: header footer_offset={}, wal_offset={}, wal_size={}",
header.footer_offset, header.wal_offset, header.wal_size
header.footer_offset,
header.wal_offset,
header.wal_size
);
println!("doctor: attempting read_toc");
doctor_log!("doctor: attempting read_toc");
let (toc, toc_offset, recovered) = match read_toc(&mut file, header) {
Ok(toc) => (toc, header.footer_offset, false),
Err(_) => match recover_toc(&mut file, Some(header.footer_offset)) {
Ok((toc, offset)) => {
println!("doctor: recover_toc succeeded at offset {}", offset);
doctor_log!("doctor: recover_toc succeeded at offset {}", offset);
probe.findings.push(DoctorFinding::warning(
DoctorFindingCode::TocDecodeFailure,
"recovered toc from trailer",
@@ -378,7 +413,7 @@ impl DoctorPlanner {
(toc, offset, true)
}
Err(err) => {
println!("doctor: recover_toc failed: {}", err);
doctor_log!("doctor: recover_toc failed: {}", err);
probe.findings.push(DoctorFinding::error(
DoctorFindingCode::TocDecodeFailure,
err.to_string(),
@@ -390,7 +425,7 @@ impl DoctorPlanner {
probe.toc_recovered = recovered;
probe.toc = Some(toc.clone());
probe.toc_offset = Some(toc_offset);
println!(
doctor_log!(
"doctor: toc entries frames={}, segments={}, segment_catalog(lex={}, vec={}, time={})",
toc.frames.len(),
toc.segments.len(),
@@ -425,7 +460,7 @@ impl DoctorPlanner {
match EmbeddedWal::open(&file, header) {
Ok(mut wal) => {
println!("doctor: embedded wal open success");
doctor_log!("doctor: embedded wal open success");
let stats = wal.stats();
probe.wal_from_sequence = header.wal_sequence;
probe.wal_to_sequence = stats.sequence;
@@ -467,9 +502,11 @@ impl DoctorPlanner {
return;
};
if let Some(manifest) = toc.time_index.clone() {
println!(
doctor_log!(
"doctor: inspect_time_index offset={} length={} entries={}",
manifest.bytes_offset, manifest.bytes_length, manifest.entry_count
manifest.bytes_offset,
manifest.bytes_length,
manifest.entry_count
);
let span_end = manifest.bytes_offset.saturating_add(manifest.bytes_length);
if span_end > probe.file_len {
@@ -493,7 +530,7 @@ impl DoctorPlanner {
}
match read_track(file, manifest.bytes_offset, manifest.bytes_length) {
Ok(entries) => {
println!("doctor: time index read {} entries", entries.len());
doctor_log!("doctor: time index read {} entries", entries.len());
probe.index.time_expected_entries = manifest.entry_count;
if entries.len() as u64 != manifest.entry_count {
probe.index.needs_time = true;
@@ -516,7 +553,7 @@ impl DoctorPlanner {
}
}
Err(err) => {
println!("doctor: read_track failed: {}", err);
doctor_log!("doctor: read_track failed: {}", err);
probe.index.needs_time = true;
probe.findings.push(DoctorFinding::warning(
DoctorFindingCode::TimeIndexChecksumMismatch,
@@ -541,7 +578,7 @@ impl DoctorPlanner {
// CRITICAL FIX (Bug #10): Check for Tantivy segments first
// Tantivy indexes have NO manifest but have lex_segments instead
if !toc.indexes.lex_segments.is_empty() {
println!(
doctor_log!(
"doctor: detected Tantivy-based lex index with {} segments, skipping old validation",
toc.indexes.lex_segments.len()
);
@@ -559,9 +596,11 @@ impl DoctorPlanner {
#[cfg(feature = "lex")]
{
println!(
doctor_log!(
"doctor: inspect_lex_index offset={} length={} docs={}",
manifest.bytes_offset, manifest.bytes_length, manifest.doc_count
manifest.bytes_offset,
manifest.bytes_length,
manifest.doc_count
);
let span_end = manifest.bytes_offset.saturating_add(manifest.bytes_length);
@@ -581,6 +620,7 @@ impl DoctorPlanner {
));
return;
}
#[allow(clippy::cast_possible_truncation)]
let mut buf = vec![0u8; manifest.bytes_length as usize];
if let Err(err) = file.seek(SeekFrom::Start(manifest.bytes_offset)) {
probe.index.needs_lex = true;
@@ -602,9 +642,10 @@ impl DoctorPlanner {
Ok(mut index) => {
let doc_count = index.documents_mut().len() as u64;
if doc_count != manifest.doc_count {
println!(
doctor_log!(
"doctor: lex doc count mismatch manifest {} actual {}",
manifest.doc_count, doc_count
manifest.doc_count,
doc_count
);
probe.index.needs_lex = true;
probe.findings.push(DoctorFinding::warning(
@@ -652,10 +693,9 @@ impl DoctorPlanner {
.segment_catalog
.vec_segments
.first()
.map(|s| s.dimension)
.unwrap_or(0);
.map_or(0, |s| s.dimension);
println!(
doctor_log!(
"doctor: inspect_vec_index (segment_catalog) segments={} total_vectors={} dim={}",
toc.segment_catalog.vec_segments.len(),
total_vectors,
@@ -684,18 +724,19 @@ impl DoctorPlanner {
probe.index.needs_vec = true;
probe.findings.push(DoctorFinding::warning(
DoctorFindingCode::VecIndexCorrupt,
format!("vec segment {} exceeds safety limit", i),
format!("vec segment {i} exceeds safety limit"),
));
continue;
}
// Read and validate segment
#[allow(clippy::cast_possible_truncation)]
let mut buf = vec![0u8; segment.common.bytes_length as usize];
if let Err(err) = file.seek(SeekFrom::Start(segment.common.bytes_offset)) {
probe.index.needs_vec = true;
probe.findings.push(DoctorFinding::warning(
DoctorFindingCode::VecIndexCorrupt,
format!("vec segment {} seek error: {}", i, err),
format!("vec segment {i} seek error: {err}"),
));
continue;
}
@@ -703,7 +744,7 @@ impl DoctorPlanner {
probe.index.needs_vec = true;
probe.findings.push(DoctorFinding::warning(
DoctorFindingCode::VecIndexCorrupt,
format!("vec segment {} read error: {}", i, err),
format!("vec segment {i} read error: {err}"),
));
continue;
}
@@ -713,7 +754,7 @@ impl DoctorPlanner {
probe.index.needs_vec = true;
probe.findings.push(DoctorFinding::warning(
DoctorFindingCode::VecIndexCorrupt,
format!("vec segment {} decode error: {}", i, err),
format!("vec segment {i} decode error: {err}"),
));
}
}
@@ -734,9 +775,12 @@ impl DoctorPlanner {
};
probe.index.vec_expected_vectors = manifest.vector_count;
probe.index.vec_dimension = manifest.dimension;
println!(
doctor_log!(
"doctor: inspect_vec_index (monolithic) offset={} length={} vectors={} dim={}",
manifest.bytes_offset, manifest.bytes_length, manifest.vector_count, manifest.dimension
manifest.bytes_offset,
manifest.bytes_length,
manifest.vector_count,
manifest.dimension
);
let span_end = manifest.bytes_offset.saturating_add(manifest.bytes_length);
@@ -758,6 +802,7 @@ impl DoctorPlanner {
return;
}
#[allow(clippy::cast_possible_truncation)]
let mut buf = vec![0u8; manifest.bytes_length as usize];
if let Err(err) = file.seek(SeekFrom::Start(manifest.bytes_offset)) {
probe.index.needs_vec = true;
@@ -807,7 +852,7 @@ impl DoctorExecutor {
}
fn run(self) -> Result<DoctorReport> {
println!("doctor: starting executor");
doctor_log!("doctor: starting executor");
let DoctorExecutor { path, plan } = self;
let mut metrics = DoctorMetrics::default();
let mut phase_reports = Vec::new();
@@ -832,7 +877,7 @@ impl DoctorExecutor {
});
}
println!("doctor: trying to open memory");
doctor_log!("doctor: trying to open memory");
// Check if WAL is corrupted - if so, attempt recovery
let has_wal_corruption = plan
@@ -841,10 +886,10 @@ impl DoctorExecutor {
.any(|f| matches!(f.code, DoctorFindingCode::WalChecksumMismatch));
let mut mem = if has_wal_corruption {
println!("doctor: WAL corrupted, attempting recovery by rebuilding WAL");
doctor_log!("doctor: WAL corrupted, attempting recovery by rebuilding WAL");
match try_recover_from_wal_corruption(&path) {
Ok(recovered_mem) => {
println!("doctor: successfully recovered from WAL corruption");
doctor_log!("doctor: successfully recovered from WAL corruption");
additional_findings.push(DoctorFinding::warning(
DoctorFindingCode::WalChecksumMismatch,
"WAL was corrupted but successfully rebuilt".to_string(),
@@ -852,10 +897,10 @@ impl DoctorExecutor {
Some(recovered_mem)
}
Err(err) => {
println!("doctor: WAL recovery failed: {}", err);
doctor_log!("doctor: WAL recovery failed: {}", err);
additional_findings.push(DoctorFinding::error(
DoctorFindingCode::WalChecksumMismatch,
format!("WAL corrupted and recovery failed: {}", err),
format!("WAL corrupted and recovery failed: {err}"),
));
return Ok(DoctorReport {
plan,
@@ -874,12 +919,12 @@ impl DoctorExecutor {
Err(err) => {
// Check if this is TOC/header corruption that aggressive repair can fix
if Self::is_toc_corruption_error(&err) {
println!("doctor: file unopenable due to TOC/header corruption");
println!("doctor: attempting aggressive repair (Tier 2)");
doctor_log!("doctor: file unopenable due to TOC/header corruption");
doctor_log!("doctor: attempting aggressive repair (Tier 2)");
match Self::aggressive_header_repair(&path) {
Ok(()) => {
println!("doctor: aggressive repair successful, retrying open");
doctor_log!("doctor: aggressive repair successful, retrying open");
additional_findings.push(DoctorFinding::warning(
DoctorFindingCode::HeaderFooterOffsetMismatch,
"Header footer_offset was corrupted but repaired via aggressive scan".to_string(),
@@ -889,13 +934,13 @@ impl DoctorExecutor {
match Memvid::try_open(&path) {
Ok(mem) => Some(mem),
Err(retry_err) => {
println!(
doctor_log!(
"doctor: file still unopenable after aggressive repair: {}",
retry_err
);
additional_findings.push(DoctorFinding::error(
DoctorFindingCode::InternalError,
format!("Aggressive repair succeeded but file still corrupt: {}", retry_err),
format!("Aggressive repair succeeded but file still corrupt: {retry_err}"),
));
return Ok(DoctorReport {
plan,
@@ -909,10 +954,10 @@ impl DoctorExecutor {
}
}
Err(repair_err) => {
println!("doctor: aggressive repair failed: {}", repair_err);
doctor_log!("doctor: aggressive repair failed: {}", repair_err);
additional_findings.push(DoctorFinding::error(
DoctorFindingCode::InternalError,
format!("Aggressive repair failed: {}", repair_err),
format!("Aggressive repair failed: {repair_err}"),
));
return Ok(DoctorReport {
plan,
@@ -951,12 +996,12 @@ impl DoctorExecutor {
let start = Instant::now();
for phase in &plan.phases {
println!("doctor: entering phase {:?}", phase.phase);
doctor_log!("doctor: entering phase {:?}", phase.phase);
let phase_start = Instant::now();
let mut actions = Vec::new();
let mut phase_status = DoctorPhaseStatus::Skipped;
for action in &phase.actions {
println!("doctor: executing action {:?}", action.action);
doctor_log!("doctor: executing action {:?}", action.action);
if mem.is_none() {
overall_failed = true;
phase_status = DoctorPhaseStatus::Failed;
@@ -1098,15 +1143,15 @@ impl DoctorExecutor {
// CRITICAL: Clear WAL before verification
// This ensures a clean slate even if doctor's own operations had WAL issues
if let Some(ref mut held) = mem {
println!("doctor: performing final WAL cleanup before verification");
doctor_log!("doctor: performing final WAL cleanup before verification");
if let Err(err) = Self::reset_wal(held) {
println!("doctor: WARNING - final WAL cleanup failed: {}", err);
doctor_log!("doctor: WARNING - final WAL cleanup failed: {}", err);
additional_findings.push(DoctorFinding::warning(
DoctorFindingCode::InternalError,
format!("final WAL cleanup failed: {}", err),
format!("final WAL cleanup failed: {err}"),
));
} else {
println!("doctor: final WAL cleanup successful");
doctor_log!("doctor: final WAL cleanup successful");
}
}
@@ -1184,7 +1229,11 @@ impl DoctorExecutor {
metrics.phase_durations.push(DoctorPhaseDuration {
phase: phase.phase,
duration_ms: phase_start.elapsed().as_millis() as u64,
duration_ms: phase_start
.elapsed()
.as_millis()
.try_into()
.unwrap_or(u64::MAX),
});
metrics.actions_completed += actions
.iter()
@@ -1210,7 +1259,7 @@ impl DoctorExecutor {
}
}
metrics.total_duration_ms = start.elapsed().as_millis() as u64;
metrics.total_duration_ms = start.elapsed().as_millis().try_into().unwrap_or(u64::MAX);
if overall_failed {
if let Some(original) = &original_header {
@@ -1372,9 +1421,11 @@ impl DoctorExecutor {
lex: bool,
vec: bool,
) -> Result<DoctorActionReport> {
println!(
doctor_log!(
"doctor: apply pending rebuilds (time={}, lex={}, vec={})",
time, lex, vec
time,
lex,
vec
);
if !(time || lex || vec) {
return Ok(DoctorActionReport {
@@ -1399,20 +1450,20 @@ impl DoctorExecutor {
// Otherwise build_vec_artifact reads from self.vec_index (which is None)
// and the vectors are lost.
if mem.vec_index.is_none() && mem.toc.indexes.vec.is_some() {
println!("doctor: loading existing vec index to preserve it");
doctor_log!("doctor: loading existing vec index to preserve it");
if let Err(e) = mem.ensure_vec_index() {
println!("doctor: warning: failed to load vec index: {e}");
doctor_log!("doctor: warning: failed to load vec index: {e}");
}
}
}
println!("doctor: rebuild_indexes start");
mem.rebuild_indexes(&[])?;
println!("doctor: rebuild_indexes done");
doctor_log!("doctor: rebuild_indexes start");
mem.rebuild_indexes(&[], &[])?;
doctor_log!("doctor: rebuild_indexes done");
// Preserve footer_offset that was just set by rebuild_indexes
let footer_offset_after_rebuild = mem.header.footer_offset;
println!(
doctor_log!(
"doctor: footer_offset after rebuild: {}",
footer_offset_after_rebuild
);
@@ -1436,7 +1487,7 @@ impl DoctorExecutor {
reason: "footer_offset corrupted during doctor repair".into(),
});
}
println!(
doctor_log!(
"doctor: footer_offset preserved: {}",
mem.header.footer_offset
);
@@ -1449,23 +1500,24 @@ impl DoctorExecutor {
}
fn reset_wal(mem: &mut Memvid) -> Result<()> {
println!(
doctor_log!(
"doctor: reset_wal - zeroing {} bytes at offset {}",
mem.header.wal_size, mem.header.wal_offset
mem.header.wal_size,
mem.header.wal_offset
);
let mut remaining = mem.header.wal_size;
let mut offset = mem.header.wal_offset;
let chunk_size = min(remaining as usize, 4096).max(1);
let chunk_size = (remaining.min(4096) as usize).max(1);
let zeros = vec![0u8; chunk_size];
while remaining > 0 {
let write_len = min(remaining as usize, zeros.len());
let write_len = usize::try_from(remaining.min(zeros.len() as u64)).unwrap_or(0);
mem.file.seek(SeekFrom::Start(offset))?;
mem.file.write_all(&zeros[..write_len])?;
remaining -= write_len as u64;
offset += write_len as u64;
}
mem.file.sync_all()?;
println!("doctor: reset_wal - WAL region zeroed and synced");
doctor_log!("doctor: reset_wal - WAL region zeroed and synced");
// CRITICAL: Update and persist header BEFORE reopening WAL
// EmbeddedWal::open will read the header, so it must have the correct values
@@ -1473,11 +1525,11 @@ impl DoctorExecutor {
mem.header.wal_sequence = 0;
crate::persist_header(&mut mem.file, &mem.header)?;
mem.file.sync_all()?;
println!("doctor: reset_wal - header updated with wal_sequence=0, wal_checkpoint_pos=0");
doctor_log!("doctor: reset_wal - header updated with wal_sequence=0, wal_checkpoint_pos=0");
// Now reopen the WAL with the clean state
mem.wal = EmbeddedWal::open(&mem.file, &mem.header)?;
println!("doctor: reset_wal - WAL reopened successfully");
doctor_log!("doctor: reset_wal - WAL reopened successfully");
// CRITICAL: Clear dirty flag to prevent Drop from calling commit()
// Drop handler will call commit() if dirty=true, which would corrupt the WAL we just cleaned
@@ -1486,7 +1538,7 @@ impl DoctorExecutor {
{
mem.tantivy_dirty = false;
}
println!("doctor: reset_wal - cleared dirty flags");
doctor_log!("doctor: reset_wal - cleared dirty flags");
Ok(())
}
@@ -1516,7 +1568,7 @@ impl DoctorExecutor {
fn scan_for_footer(path: &Path) -> Result<u64> {
use std::fs::File;
println!("doctor: [Tier 2] Scanning for footer in corrupted file");
doctor_log!("doctor: [Tier 2] Scanning for footer in corrupted file");
let file = File::open(path)?;
let file_size = file.metadata()?.len();
@@ -1538,15 +1590,15 @@ impl DoctorExecutor {
let mut buf = [0u8; 8];
reader.read_exact(&mut buf)?;
if &buf == FOOTER_MAGIC {
println!(
if buf == FOOTER_MAGIC {
doctor_log!(
"doctor: [Tier 2] Footer found at expected location: {}",
expected_offset
);
return Ok(expected_offset);
}
println!("doctor: [Tier 2] Footer not at expected location, scanning backwards...");
doctor_log!("doctor: [Tier 2] Footer not at expected location, scanning backwards...");
// Scan backwards from near end of file
const MAX_SCAN: u64 = 100_000_000; // Scan last 100MB max
@@ -1555,14 +1607,14 @@ impl DoctorExecutor {
for offset in (scan_start..file_size.saturating_sub(FOOTER_SIZE)).rev() {
reader.seek(SeekFrom::Start(offset))?;
reader.read_exact(&mut buf)?;
if &buf == FOOTER_MAGIC {
println!("doctor: [Tier 2] Footer found at offset: {}", offset);
if buf == FOOTER_MAGIC {
doctor_log!("doctor: [Tier 2] Footer found at offset: {}", offset);
return Ok(offset);
}
// Progress indicator every 10MB
if offset % 10_000_000 == 0 {
println!("doctor: [Tier 2] Scanned to offset {}...", offset);
doctor_log!("doctor: [Tier 2] Scanned to offset {}...", offset);
}
}
@@ -1571,9 +1623,9 @@ impl DoctorExecutor {
})
}
/// Tier 2 Aggressive Repair: Fix header's footer_offset pointer.
/// Tier 2 Aggressive Repair: Fix header's `footer_offset` pointer.
fn aggressive_header_repair(path: &Path) -> Result<()> {
println!("doctor: [Tier 2] Attempting aggressive header repair");
doctor_log!("doctor: [Tier 2] Attempting aggressive header repair");
// Find actual footer location
let actual_footer_offset = Self::scan_for_footer(path)?;
@@ -1585,27 +1637,23 @@ impl DoctorExecutor {
file.read_exact(&mut buf)?;
let header_footer_offset = u64::from_le_bytes(buf);
println!(
doctor_log!(
"doctor: [Tier 2] Header claims footer at: {}",
header_footer_offset
);
println!(
doctor_log!(
"doctor: [Tier 2] Actual footer at: {}",
actual_footer_offset
);
if header_footer_offset == actual_footer_offset {
println!("doctor: [Tier 2] Header already correct");
doctor_log!("doctor: [Tier 2] Header already correct");
return Ok(());
}
// Fix header
let mismatch = if actual_footer_offset > header_footer_offset {
actual_footer_offset - header_footer_offset
} else {
header_footer_offset - actual_footer_offset
};
println!(
let mismatch = actual_footer_offset.abs_diff(header_footer_offset);
doctor_log!(
"doctor: [Tier 2] Mismatch: {} bytes, repairing...",
mismatch
);
@@ -1620,7 +1668,7 @@ impl DoctorExecutor {
let new_value = u64::from_le_bytes(buf);
if new_value == actual_footer_offset {
println!("doctor: [Tier 2] Header repaired successfully");
doctor_log!("doctor: [Tier 2] Header repaired successfully");
Ok(())
} else {
Err(MemvidError::InvalidHeader {
+23 -14
View File
@@ -30,6 +30,7 @@ pub struct EnrichmentHandle {
impl EnrichmentHandle {
/// Stop the worker and wait for it to finish.
#[must_use]
pub fn stop_and_wait(mut self) -> EnrichmentWorkerStats {
self.handle.stop();
if let Some(thread) = self.thread.take() {
@@ -39,11 +40,13 @@ impl EnrichmentHandle {
}
/// Check if worker is still running.
#[must_use]
pub fn is_running(&self) -> bool {
self.handle.is_running()
}
/// Get current statistics.
#[must_use]
pub fn stats(&self) -> EnrichmentWorkerStats {
self.handle.stats()
}
@@ -213,16 +216,19 @@ where
impl Memvid {
/// Get the number of frames pending enrichment.
#[must_use]
pub fn enrichment_queue_len(&self) -> usize {
self.toc.enrichment_queue.len()
}
/// Check if any frames need enrichment.
#[must_use]
pub fn has_pending_enrichment(&self) -> bool {
!self.toc.enrichment_queue.is_empty()
}
/// Get the next task from the enrichment queue.
#[must_use]
pub fn next_enrichment_task(&self) -> Option<EnrichmentTask> {
self.toc.enrichment_queue.tasks.first().cloned()
}
@@ -235,7 +241,8 @@ impl Memvid {
/// Read frame data needed for enrichment.
///
/// Returns (search_text, is_skim, needs_embedding) if frame exists.
/// Returns (`search_text`, `is_skim`, `needs_embedding`) if frame exists.
#[must_use]
pub fn read_frame_for_enrichment(&self, frame_id: FrameId) -> Option<(String, bool, bool)> {
let frame = self
.toc
@@ -249,8 +256,7 @@ impl Memvid {
let is_skim = frame
.extra_metadata
.get("skim")
.map(|v| v == "true")
.unwrap_or(false);
.is_some_and(|v| v == "true");
// Check if embeddings are needed
let needs_embedding = frame.enrichment_state == EnrichmentState::Searchable;
@@ -269,7 +275,7 @@ impl Memvid {
.iter()
.find(|f| f.id == frame_id && f.status == FrameStatus::Active)
.cloned()
.ok_or_else(|| crate::MemvidError::FrameNotFound { frame_id })?;
.ok_or(crate::MemvidError::FrameNotFound { frame_id })?;
// Read the payload
let payload = self.read_frame_payload_bytes(&frame)?;
@@ -303,7 +309,7 @@ impl Memvid {
.frames
.iter()
.find(|f| f.id == frame_id && f.status == FrameStatus::Active)
.ok_or_else(|| crate::MemvidError::FrameNotFound { frame_id })?
.ok_or(crate::MemvidError::FrameNotFound { frame_id })?
.clone();
// Delete old document
@@ -395,7 +401,7 @@ impl Memvid {
// Mark frame as enriched
self.mark_frame_enriched(task.frame_id);
result.elapsed_ms = start.elapsed().as_millis() as u64;
result.elapsed_ms = start.elapsed().as_millis().try_into().unwrap_or(u64::MAX);
result
}
@@ -431,6 +437,7 @@ impl Memvid {
}
/// Get enrichment statistics.
#[must_use]
pub fn enrichment_stats(&self) -> EnrichmentStats {
let total_frames = self
.toc
@@ -503,6 +510,7 @@ impl Memvid {
bytes_length: artifact.bytes.len() as u64,
checksum: artifact.checksum,
compression_mode: crate::types::VectorCompression::None,
model: self.vec_model.clone(),
});
self.dirty = true;
@@ -526,7 +534,7 @@ impl Memvid {
/// 3. Updates indexes
/// 4. Marks frames as enriched
///
/// Returns (frames_processed, embeddings_generated).
/// Returns (`frames_processed`, `embeddings_generated`).
pub fn process_enrichment_with_embeddings<E: VecEmbedder>(
&mut self,
embedder: E,
@@ -595,7 +603,7 @@ impl Memvid {
frames_processed += 1;
// Update checkpoint in queue for crash recovery
let chunks_done = embeddings_generated as u32;
let chunks_done = u32::try_from(embeddings_generated).unwrap_or(u32::MAX);
self.toc.enrichment_queue.update_checkpoint(
task.frame_id,
chunks_done,
@@ -631,18 +639,19 @@ impl Memvid {
}
/// Check if vector embeddings are enabled.
#[must_use]
pub fn has_embeddings(&self) -> bool {
self.vec_enabled && self.vec_index.is_some()
}
/// Get vector count from the index.
#[must_use]
pub fn vector_count(&self) -> usize {
self.toc
.indexes
.vec
.as_ref()
.map(|m| m.vector_count as usize)
.unwrap_or(0)
self.toc.indexes.vec.as_ref().map_or(0, |m| {
#[allow(clippy::cast_possible_truncation)]
let count = m.vector_count as usize;
count
})
}
}
+82 -51
View File
@@ -70,6 +70,7 @@ impl Read for BlobReader {
if remaining == 0 {
return Ok(0);
}
#[allow(clippy::cast_possible_truncation)]
let to_read = remaining.min(buf.len() as u64) as usize;
file.seek(SeekFrom::Start(*start + *pos))?;
let read = file.read(&mut buf[..to_read])?;
@@ -161,9 +162,11 @@ fn mime_is_text(mime: &str) -> bool {
impl Memvid {
pub fn frame_by_id(&self, frame_id: FrameId) -> Result<Frame> {
let index =
usize::try_from(frame_id).map_err(|_| MemvidError::FrameNotFound { frame_id })?;
self.toc
.frames
.get(frame_id as usize)
.get(index)
.cloned()
.ok_or(MemvidError::FrameNotFound { frame_id })
}
@@ -178,8 +181,7 @@ impl Memvid {
frame
.uri
.as_deref()
.map(|candidate_uri| candidate_uri == uri)
.unwrap_or(false)
.is_some_and(|candidate_uri| candidate_uri == uri)
&& frame.status == FrameStatus::Active
})
.or_else(|| {
@@ -202,6 +204,7 @@ impl Memvid {
/// we can skip re-ingestion. The hash is computed from the original file bytes.
///
/// Returns `None` if no matching frame is found.
#[must_use]
pub fn find_frame_by_hash(&self, hash: &[u8; 32]) -> Option<&Frame> {
self.toc
.frames
@@ -254,11 +257,17 @@ impl Memvid {
}
pub fn frame_preview_by_id(&mut self, frame_id: FrameId) -> Result<String> {
let frame = self.toc.frames.get(frame_id as usize).cloned().ok_or(
MemvidError::InvalidTimeIndex {
let index = usize::try_from(frame_id).map_err(|_| MemvidError::InvalidTimeIndex {
reason: "frame id too large".into(),
})?;
let frame = self
.toc
.frames
.get(index)
.cloned()
.ok_or(MemvidError::InvalidTimeIndex {
reason: "frame id out of range".into(),
},
)?;
})?;
self.frame_preview(&frame)
}
@@ -267,11 +276,17 @@ impl Memvid {
/// Unlike `frame_preview_by_id` which truncates for display purposes,
/// this returns the complete text content suitable for LLM processing.
pub fn frame_text_by_id(&mut self, frame_id: FrameId) -> Result<String> {
let frame = self.toc.frames.get(frame_id as usize).cloned().ok_or(
MemvidError::InvalidTimeIndex {
let index = usize::try_from(frame_id).map_err(|_| MemvidError::InvalidTimeIndex {
reason: "frame id too large".into(),
})?;
let frame = self
.toc
.frames
.get(index)
.cloned()
.ok_or(MemvidError::InvalidTimeIndex {
reason: "frame id out of range".into(),
},
)?;
})?;
self.frame_content(&frame)
}
@@ -279,7 +294,7 @@ impl Memvid {
if let Some(text) = frame
.metadata
.as_ref()
.and_then(|meta| crate::image_preview_from_metadata(meta))
.and_then(crate::image_preview_from_metadata)
{
return Ok(text);
}
@@ -294,8 +309,8 @@ impl Memvid {
let label = video
.filename
.as_deref()
.or_else(|| frame.title.as_deref())
.or_else(|| frame.uri.as_deref())
.or(frame.title.as_deref())
.or(frame.uri.as_deref())
.unwrap_or("video");
segments.push(format!("Video: {label}"));
if let Some(duration) = video.duration_ms {
@@ -303,7 +318,7 @@ impl Memvid {
segments.push(format!("{seconds:.1}s"));
}
if let (Some(width), Some(height)) = (video.width, video.height) {
segments.push(format!("{}x{}", width, height));
segments.push(format!("{width}x{height}"));
}
if let Some(codec) = &video.codec {
if !codec.trim().is_empty() {
@@ -326,6 +341,13 @@ impl Memvid {
}
pub(crate) fn frame_content(&mut self, frame: &Frame) -> Result<String> {
// Check search_text first - this handles no_raw mode where payload is empty
// but search_text contains the indexed content
if let Some(search) = &frame.search_text {
if !search.is_empty() {
return Ok(search.clone());
}
}
if frame.payload_length == 0 && frame.chunk_manifest.is_none() {
return Ok(String::new());
}
@@ -337,40 +359,43 @@ impl Memvid {
return Ok(None);
}
self.ensure_vec_index()?;
Ok(self.vec_index.as_ref().and_then(|index| {
index
.embedding_for(frame_id)
.map(|embedding| embedding.to_vec())
}))
Ok(self
.vec_index
.as_ref()
.and_then(|index| index.embedding_for(frame_id).map(<[f32]>::to_vec)))
}
pub fn frame_context(&mut self, frame_id: FrameId, query: &str) -> Result<(String, usize)> {
let frame = self.toc.frames.get(frame_id as usize).cloned().ok_or(
MemvidError::InvalidTimeIndex {
let index = usize::try_from(frame_id).map_err(|_| MemvidError::InvalidTimeIndex {
reason: "frame id too large".into(),
})?;
let frame = self
.toc
.frames
.get(index)
.cloned()
.ok_or(MemvidError::InvalidTimeIndex {
reason: "frame id out of range".into(),
},
)?;
})?;
let preview = self.frame_preview(&frame)?;
let content = self.frame_content(&frame)?;
let count = query
.split_whitespace()
.filter(|token| !token.is_empty())
.map(|needle| needle.to_lowercase())
.map(str::to_lowercase)
.map(|needle| content.to_lowercase().matches(&needle).count())
.sum();
Ok((preview, count))
}
pub(crate) fn frame_canonical_bytes(&mut self, frame: &Frame) -> Result<Vec<u8>> {
if frame.role == FrameRole::Document {
if frame.chunk_manifest.is_some() {
let chunks = self.document_chunk_payloads(frame)?;
let mut buffer = Vec::new();
for (_, bytes) in chunks {
buffer.extend_from_slice(&bytes);
}
return Ok(buffer);
if frame.role == FrameRole::Document && frame.chunk_manifest.is_some() {
let chunks = self.document_chunk_payloads(frame)?;
let mut buffer = Vec::new();
for (_, bytes) in chunks {
buffer.extend_from_slice(&bytes);
}
return Ok(buffer);
}
let raw = self.read_frame_payload_bytes(frame)?;
let decoded = crate::decode_canonical_bytes(&raw, frame.canonical_encoding, frame.id)?;
@@ -409,8 +434,9 @@ impl Memvid {
if !mime_is_text(meta) {
let logical = frame
.canonical_length
.or_else(|| Some(frame.payload_length))
.or(Some(frame.payload_length))
.unwrap_or(frame.payload_length);
#[allow(clippy::cast_possible_truncation)]
return Ok(Self::render_binary_summary(logical as usize));
}
}
@@ -530,24 +556,27 @@ impl Memvid {
FrameRole::DocumentChunk => {
// Try to resolve via parent's chunk manifest (new format)
if let Some(parent_id) = frame.parent_id {
if let Some(parent) = self.toc.frames.get(parent_id as usize).cloned() {
if parent.chunk_manifest.is_some() {
if let Ok(payloads) = self.document_chunk_payloads(&parent) {
if let Some(idx) = frame.chunk_index {
let idx = idx as usize;
if idx < payloads.len() {
let mut offset = 0usize;
for (_, bytes) in payloads.iter().take(idx) {
offset += bytes.len();
// Safe frame lookup
if let Ok(index) = usize::try_from(parent_id) {
if let Some(parent) = self.toc.frames.get(index).cloned() {
if parent.chunk_manifest.is_some() {
if let Ok(payloads) = self.document_chunk_payloads(&parent) {
if let Some(idx) = frame.chunk_index {
let idx = idx as usize;
if idx < payloads.len() {
let mut offset = 0usize;
for (_, bytes) in payloads.iter().take(idx) {
offset += bytes.len();
}
let (_, bytes) = &payloads[idx];
let text = String::from_utf8_lossy(bytes).into_owned();
let end = offset + bytes.len();
return Ok(ChunkInfo {
start: offset,
end,
text,
});
}
let (_, bytes) = &payloads[idx];
let text = String::from_utf8_lossy(bytes).into_owned();
let end = offset + bytes.len();
return Ok(ChunkInfo {
start: offset,
end,
text,
});
}
}
}
@@ -589,6 +618,8 @@ impl Memvid {
pub(crate) fn read_frame_payload_bytes(&mut self, frame: &Frame) -> Result<Vec<u8>> {
self.validate_frame_bounds(frame)?;
self.file.seek(SeekFrom::Start(frame.payload_offset))?;
// Safe: guarded by MAX_FRAME_BYTES check
#[allow(clippy::cast_possible_truncation)]
let mut buf = vec![0u8; frame.payload_length as usize];
self.file.read_exact(&mut buf)?;
Ok(buf)
+5 -3
View File
@@ -1,3 +1,5 @@
// Safe expect: guaranteed non-empty iterators after length check.
#![allow(clippy::expect_used)]
use std::collections::HashMap;
use crate::memvid::lifecycle::Memvid;
@@ -11,6 +13,7 @@ use crate::{MemvidError, Result};
impl Memvid {
/// Returns the vector index dimension stored in the MV2 file, if available.
/// This is useful for auto-detecting which embedding model was used to create the file.
#[must_use]
pub fn vec_index_dimension(&self) -> Option<u32> {
self.toc
.indexes
@@ -61,8 +64,7 @@ impl Memvid {
(Some(manifest), Some(segment)) if manifest != segment => {
Err(MemvidError::InvalidToc {
reason: format!(
"vector dimension mismatch between manifest ({}) and segment catalog ({})",
manifest, segment
"vector dimension mismatch between manifest ({manifest}) and segment catalog ({segment})"
)
.into(),
})
@@ -130,7 +132,7 @@ impl Memvid {
if len == 0 {
"<binary payload: 0 bytes>".into()
} else {
format!("<binary payload: {} bytes>", len)
format!("<binary payload: {len} bytes>")
}
}
+117 -36
View File
@@ -28,8 +28,8 @@ use crate::types::FrameId;
#[cfg(feature = "parallel_segments")]
use crate::types::IndexSegmentRef;
use crate::types::{
FrameStatus, Header, IndexManifests, LogicMesh, MemoriesTrack, SchemaRegistry, SegmentCatalog,
SketchTrack, TicketRef, Tier, Toc, VectorCompression,
FrameStatus, Header, IndexManifests, LogicMesh, MemoriesTrack, PutManyOpts, SchemaRegistry,
SegmentCatalog, SketchTrack, TicketRef, Tier, Toc, VectorCompression,
};
#[cfg(feature = "temporal_track")]
use crate::{TemporalTrack, temporal_track_read};
@@ -58,6 +58,9 @@ pub struct Memvid {
/// This lets frontends predict stable frame IDs before an explicit commit.
pub(crate) pending_frame_inserts: u64,
pub(crate) data_end: u64,
/// Cached end of the payload region (max of payload_offset + payload_length across all frames).
/// Updated incrementally on frame insert to avoid O(n) scans.
pub(crate) cached_payload_end: u64,
pub(crate) generation: u64,
pub(crate) lock_settings: LockSettings,
pub(crate) lex_enabled: bool,
@@ -67,6 +70,7 @@ pub struct Memvid {
pub(crate) lex_storage: Arc<RwLock<EmbeddedLexStorage>>,
pub(crate) vec_enabled: bool,
pub(crate) vec_compression: VectorCompression,
pub(crate) vec_model: Option<String>,
pub(crate) vec_index: Option<VecIndex>,
/// CLIP visual embeddings index (separate from vec due to different dimensions)
pub(crate) clip_enabled: bool,
@@ -90,6 +94,8 @@ pub struct Memvid {
pub(crate) schema_registry: SchemaRegistry,
/// Whether to enforce strict schema validation on card insert.
pub(crate) schema_strict: bool,
/// Active batch mode options (set by `begin_batch`, cleared by `end_batch`).
pub(crate) batch_opts: Option<PutManyOpts>,
/// Active replay session being recorded (if any).
#[cfg(feature = "replay")]
pub(crate) active_session: Option<crate::replay::ActiveSession>,
@@ -99,7 +105,7 @@ pub struct Memvid {
}
/// Controls read-only open behaviour for `.mv2` memories.
#[derive(Debug, Clone, Copy)]
#[derive(Debug, Clone, Copy, Default)]
pub struct OpenReadOptions {
pub allow_repair: bool,
}
@@ -125,14 +131,6 @@ impl Default for LockSettings {
}
}
impl Default for OpenReadOptions {
fn default() -> Self {
Self {
allow_repair: false,
}
}
}
impl Memvid {
/// Create a new, empty `.mv2` file with an embedded WAL and empty TOC.
/// The file is locked exclusively for the lifetime of the handle.
@@ -177,6 +175,9 @@ impl Memvid {
#[cfg(feature = "parallel_segments")]
let manifest_wal_entries = manifest_wal.replay()?;
// No frames yet, so payload region ends at WAL boundary
let cached_payload_end = header.wal_offset + header.wal_size;
let mut memvid = Self {
file,
path: path_ref.to_path_buf(),
@@ -187,6 +188,7 @@ impl Memvid {
wal,
pending_frame_inserts: 0,
data_end,
cached_payload_end,
generation: 0,
lock_settings: LockSettings::default(),
lex_enabled: cfg!(feature = "lex"), // Enable by default if feature is enabled
@@ -195,6 +197,7 @@ impl Memvid {
lex_storage,
vec_enabled: cfg!(feature = "vec"), // Enable by default if feature is enabled
vec_compression: VectorCompression::None,
vec_model: None,
vec_index: None,
clip_enabled: cfg!(feature = "clip"), // Enable by default if feature is enabled
clip_index: None,
@@ -212,6 +215,7 @@ impl Memvid {
sketch_track: SketchTrack::default(),
schema_registry: SchemaRegistry::new(),
schema_strict: false,
batch_opts: None,
#[cfg(feature = "replay")]
active_session: None,
#[cfg(feature = "replay")]
@@ -251,6 +255,7 @@ impl Memvid {
bytes_length: 0,
checksum: empty_checksum,
compression_mode: memvid.vec_compression.clone(),
model: memvid.vec_model.clone(),
});
}
@@ -261,6 +266,7 @@ impl Memvid {
Ok(memvid)
}
#[must_use]
pub fn lock_settings(&self) -> &LockSettings {
&self.lock_settings
}
@@ -276,6 +282,7 @@ impl Memvid {
}
/// Get the current vector compression mode
#[must_use]
pub fn vector_compression(&self) -> &VectorCompression {
&self.vec_compression
}
@@ -285,6 +292,7 @@ impl Memvid {
/// Frame IDs are dense indices into `toc.frames`. When a memory is mutable, inserts are first
/// appended to the embedded WAL and only materialized into `toc.frames` on commit. This helper
/// lets frontends allocate stable frame IDs before an explicit commit.
#[must_use]
pub fn next_frame_id(&self) -> u64 {
(self.toc.frames.len() as u64).saturating_add(self.pending_frame_inserts)
}
@@ -313,7 +321,7 @@ impl Memvid {
let mut header = HeaderCodec::read(&mut file)?;
let toc = match read_toc(&mut file, &header) {
Ok(toc) => toc,
Err(err @ MemvidError::Decode(_)) | Err(err @ MemvidError::InvalidToc { .. }) => {
Err(err @ (MemvidError::Decode(_) | MemvidError::InvalidToc { .. })) => {
tracing::info!("toc decode failed ({}); attempting recovery", err);
let (toc, recovered_offset) = recover_toc(&mut file, Some(header.footer_offset))?;
if recovered_offset != header.footer_offset
@@ -362,6 +370,7 @@ impl Memvid {
wal,
pending_frame_inserts: 0,
data_end: 0,
cached_payload_end: 0,
generation,
lock_settings: LockSettings::default(),
lex_enabled: false,
@@ -370,6 +379,7 @@ impl Memvid {
lex_storage,
vec_enabled: false,
vec_compression: VectorCompression::None,
vec_model: None,
vec_index: None,
clip_enabled: false,
clip_index: None,
@@ -387,12 +397,15 @@ impl Memvid {
sketch_track: SketchTrack::default(),
schema_registry: SchemaRegistry::new(),
schema_strict: false,
batch_opts: None,
#[cfg(feature = "replay")]
active_session: None,
#[cfg(feature = "replay")]
completed_sessions: Vec::new(),
};
memvid.data_end = compute_data_end(&memvid.toc, &memvid.header);
// One-time O(n) scan to initialize cached_payload_end from existing frames
memvid.cached_payload_end = compute_payload_region_end(&memvid.toc, &memvid.header);
// Use consolidated helper for lex_enabled check
memvid.lex_enabled = has_lex_index(&memvid.toc);
if memvid.lex_enabled {
@@ -480,6 +493,8 @@ impl Memvid {
&toc.indexes.lex_segments,
)));
let cached_payload_end = compute_payload_region_end(&toc, &header);
let mut memvid = Self {
file,
path: path_ref.to_path_buf(),
@@ -490,6 +505,7 @@ impl Memvid {
wal,
pending_frame_inserts: 0,
data_end,
cached_payload_end,
generation,
lock_settings: LockSettings::default(),
lex_enabled: false,
@@ -498,6 +514,7 @@ impl Memvid {
lex_storage,
vec_enabled: false,
vec_compression: VectorCompression::None,
vec_model: None,
vec_index: None,
clip_enabled: false,
clip_index: None,
@@ -515,6 +532,7 @@ impl Memvid {
sketch_track: SketchTrack::default(),
schema_registry: SchemaRegistry::new(),
schema_strict: false,
batch_opts: None,
#[cfg(feature = "replay")]
active_session: None,
#[cfg(feature = "replay")]
@@ -622,6 +640,13 @@ impl Memvid {
};
// Read the compressed data from the file
if manifest.bytes_length > crate::MAX_INDEX_BYTES {
return Err(MemvidError::InvalidToc {
reason: "memories track exceeds safety limit".into(),
});
}
// Safe: guarded by MAX_INDEX_BYTES check above
#[allow(clippy::cast_possible_truncation)]
let mut buf = vec![0u8; manifest.bytes_length as usize];
self.file
.seek(std::io::SeekFrom::Start(manifest.bytes_offset))?;
@@ -649,6 +674,13 @@ impl Memvid {
};
// Read the serialized data from the file
if manifest.bytes_length > crate::MAX_INDEX_BYTES {
return Err(MemvidError::InvalidToc {
reason: "logic mesh exceeds safety limit".into(),
});
}
// Safe: guarded by MAX_INDEX_BYTES check above
#[allow(clippy::cast_possible_truncation)]
let mut buf = vec![0u8; manifest.bytes_length as usize];
self.file
.seek(std::io::SeekFrom::Start(manifest.bytes_offset))?;
@@ -763,18 +795,20 @@ impl Memvid {
///
/// Returns the binding if this file is bound to a dashboard memory,
/// or None if unbound.
#[must_use]
pub fn get_memory_binding(&self) -> Option<&crate::types::MemoryBinding> {
self.toc.memory_binding.as_ref()
}
/// Bind this file to a dashboard memory.
///
/// This stores the binding in the TOC and applies the ticket for capacity.
/// The caller (CLI/bindings) is responsible for fetching the ticket from the API.
/// This stores the binding in the TOC and applies a temporary ticket for initial binding.
/// The caller should follow up with `apply_signed_ticket` for cryptographic verification.
///
/// # Errors
///
/// Returns `MemoryAlreadyBound` if this file is already bound to a different memory.
#[allow(deprecated)]
pub fn bind_memory(
&mut self,
binding: crate::types::MemoryBinding,
@@ -801,6 +835,36 @@ impl Memvid {
Ok(())
}
/// Set only the memory binding without applying a ticket.
///
/// This is used when the caller will immediately follow up with `apply_signed_ticket`
/// to apply the cryptographically verified ticket. This avoids the sequence number
/// conflict that occurs when using `bind_memory` with a temporary ticket.
///
/// # Errors
///
/// Returns `MemoryAlreadyBound` if this file is already bound to a different memory.
pub fn set_memory_binding_only(&mut self, binding: crate::types::MemoryBinding) -> Result<()> {
self.ensure_writable()?;
// Check existing binding
if let Some(existing) = self.get_memory_binding() {
if existing.memory_id != binding.memory_id {
return Err(MemvidError::MemoryAlreadyBound {
existing_memory_id: existing.memory_id,
existing_memory_name: existing.memory_name.clone(),
bound_at: existing.bound_at.to_rfc3339(),
});
}
}
// Store binding in TOC (without applying a ticket)
self.toc.memory_binding = Some(binding);
self.dirty = true;
Ok(())
}
/// Unbind this file from its dashboard memory.
///
/// This clears the binding and reverts to free tier capacity (1 GB).
@@ -812,6 +876,7 @@ impl Memvid {
seq_no: 1,
expires_in_secs: 0,
capacity_bytes: crate::types::Tier::Free.capacity_bytes(),
verified: false,
};
self.dirty = true;
Ok(())
@@ -830,7 +895,14 @@ pub(crate) fn read_toc(file: &mut File, header: &Header) -> Result<Toc> {
// Read the entire region from footer_offset to EOF (includes TOC + footer)
file.seek(SeekFrom::Start(header.footer_offset))?;
// Safe: total_size bounded by file length, and we check MAX_INDEX_BYTES before reading
#[allow(clippy::cast_possible_truncation)]
let total_size = (len - header.footer_offset) as usize;
if total_size as u64 > crate::MAX_INDEX_BYTES {
return Err(MemvidError::InvalidToc {
reason: "toc region exceeds safety limit".into(),
});
}
if total_size < FOOTER_SIZE {
return Err(MemvidError::InvalidToc {
@@ -850,6 +922,7 @@ pub(crate) fn read_toc(file: &mut File, header: &Header) -> Result<Toc> {
// Extract only the TOC bytes (excluding the footer)
let toc_bytes = &buf[..footer_start];
#[allow(clippy::cast_possible_truncation)]
if toc_bytes.len() != footer.toc_len as usize {
return Err(MemvidError::InvalidToc {
reason: "toc length mismatch".into(),
@@ -919,10 +992,10 @@ fn verify_toc_prefix(bytes: &[u8]) -> Result<()> {
/// Ensure frame payloads do not overlap each other or exceed file boundary.
///
/// Frames in the TOC are ordered by frame_id, not by payload_offset, so we must
/// sort by payload_offset before checking for overlaps.
/// Frames in the TOC are ordered by `frame_id`, not by `payload_offset`, so we must
/// sort by `payload_offset` before checking for overlaps.
///
/// Note: Frames with payload_length == 0 are "virtual" frames (e.g., document
/// Note: Frames with `payload_length` == 0 are "virtual" frames (e.g., document
/// frames that reference chunks) and are skipped from this check.
fn ensure_non_overlapping_frames(toc: &Toc, file_len: u64) -> Result<()> {
// Collect active frames with actual payloads and sort by payload_offset
@@ -994,6 +1067,8 @@ pub(crate) fn recover_toc(file: &mut File, hint: Option<u64>) -> Result<(Toc, u6
if let Some(hint_offset) = hint {
use crate::footer::FOOTER_SIZE;
// Safe: file successfully mmapped so length fits in usize
#[allow(clippy::cast_possible_truncation)]
let start = (hint_offset.min(len)) as usize;
if mmap.len().saturating_sub(start) >= FOOTER_SIZE {
let toc_end = mmap.len().saturating_sub(FOOTER_SIZE);
@@ -1017,6 +1092,8 @@ pub(crate) fn recover_toc(file: &mut File, hint: Option<u64>) -> Result<(Toc, u6
// Fallback to manual scan if footer-based recovery failed
let mut ranges = Vec::new();
if let Some(hint_offset) = hint {
// Safe: file successfully mmapped so length fits in usize
#[allow(clippy::cast_possible_truncation)]
let hint_idx = hint_offset.min(len) as usize;
ranges.push((hint_idx, mmap.len()));
if hint_idx > 0 {
@@ -1109,6 +1186,7 @@ pub(crate) fn empty_toc() -> Toc {
seq_no: 1,
expires_in_secs: 0,
capacity_bytes: Tier::Free.capacity_bytes(),
verified: false,
},
memory_binding: None,
replay_manifest: None,
@@ -1118,6 +1196,21 @@ pub(crate) fn empty_toc() -> Toc {
}
}
/// Compute the end of the payload region from frame payloads only.
/// Used once at open time to seed `cached_payload_end`.
pub(crate) fn compute_payload_region_end(toc: &Toc, header: &Header) -> u64 {
let wal_region_end = header.wal_offset.saturating_add(header.wal_size);
let mut max_end = wal_region_end;
for frame in &toc.frames {
if frame.payload_length != 0 {
if let Some(end) = frame.payload_offset.checked_add(frame.payload_length) {
max_end = max_end.max(end);
}
}
}
max_end
}
pub(crate) fn compute_data_end(toc: &Toc, header: &Header) -> u64 {
// `data_end` tracks the end of all data bytes that should not be overwritten by appends:
// - frame payloads
@@ -1327,7 +1420,7 @@ pub(crate) fn cleanup_manifest_wal_public(path: &Path) {
}
/// Single source of truth: does this TOC have a lexical index?
/// Checks all possible locations: old manifest, lex_segments, and tantivy_segments.
/// Checks all possible locations: old manifest, `lex_segments`, and `tantivy_segments`.
pub(crate) fn has_lex_index(toc: &Toc) -> bool {
toc.segment_catalog.lex_enabled
|| toc.indexes.lex.is_some()
@@ -1408,17 +1501,13 @@ fn validate_segment_integrity(toc: &Toc, header: &Header, file_len: u64) -> Resu
let end = offset
.checked_add(length)
.ok_or_else(|| MemvidError::Doctor {
reason: format!(
"Tantivy segment {} offset overflow: {} + {}",
idx, offset, length
),
reason: format!("Tantivy segment {idx} offset overflow: {offset} + {length}"),
})?;
if end > file_len || end > data_limit {
return Err(MemvidError::Doctor {
reason: format!(
"Tantivy segment {} out of bounds: offset={}, length={}, end={}, file_len={}, data_limit={}",
idx, offset, length, end, file_len, data_limit
"Tantivy segment {idx} out of bounds: offset={offset}, length={length}, end={end}, file_len={file_len}, data_limit={data_limit}"
),
});
}
@@ -1436,17 +1525,13 @@ fn validate_segment_integrity(toc: &Toc, header: &Header, file_len: u64) -> Resu
let end = offset
.checked_add(length)
.ok_or_else(|| MemvidError::Doctor {
reason: format!(
"Time segment {} offset overflow: {} + {}",
idx, offset, length
),
reason: format!("Time segment {idx} offset overflow: {offset} + {length}"),
})?;
if end > file_len || end > data_limit {
return Err(MemvidError::Doctor {
reason: format!(
"Time segment {} out of bounds: offset={}, length={}, end={}, file_len={}, data_limit={}",
idx, offset, length, end, file_len, data_limit
"Time segment {idx} out of bounds: offset={offset}, length={length}, end={end}, file_len={file_len}, data_limit={data_limit}"
),
});
}
@@ -1464,17 +1549,13 @@ fn validate_segment_integrity(toc: &Toc, header: &Header, file_len: u64) -> Resu
let end = offset
.checked_add(length)
.ok_or_else(|| MemvidError::Doctor {
reason: format!(
"Vec segment {} offset overflow: {} + {}",
idx, offset, length
),
reason: format!("Vec segment {idx} offset overflow: {offset} + {length}"),
})?;
if end > file_len || end > data_limit {
return Err(MemvidError::Doctor {
reason: format!(
"Vec segment {} out of bounds: offset={}, length={}, end={}, file_len={}, data_limit={}",
idx, offset, length, end, file_len, data_limit
"Vec segment {idx} out of bounds: offset={offset}, length={length}, end={end}, file_len={file_len}, data_limit={data_limit}"
),
});
}
+17 -13
View File
@@ -124,7 +124,7 @@ impl Memvid {
// In strict mode, reject all if any are invalid
let errors: Vec<String> = validation_errors
.iter()
.map(|(i, e)| format!("Card {}: {}", i, e))
.map(|(i, e)| format!("Card {i}: {e}"))
.collect();
return Err(crate::error::MemvidError::SchemaValidation {
reason: format!(
@@ -413,6 +413,7 @@ impl Memvid {
///
/// # Returns
/// A vector of (index, error) tuples for invalid cards.
#[must_use]
pub fn validate_cards(&self, cards: &[MemoryCard]) -> Vec<(usize, SchemaError)> {
cards
.iter()
@@ -424,7 +425,7 @@ impl Memvid {
/// Infer schemas from existing memory cards.
///
/// Analyzes all predicates (slots) in the memories track and infers
/// type information (Number, DateTime, Boolean, String) and cardinality
/// type information (Number, `DateTime`, Boolean, String) and cardinality
/// (Single vs Multiple) from the actual values.
///
/// # Returns
@@ -455,7 +456,7 @@ impl Memvid {
let all_values: Vec<&str> = entity_values
.values()
.flatten()
.map(|s| s.as_str())
.map(std::string::String::as_str)
.collect();
// Use the registry's inference method
@@ -539,16 +540,15 @@ impl Memvid {
.into_iter()
.map(|schema| {
let stats = predicate_stats.get(&schema.id);
let (entity_count, value_count, unique_values) = stats
.map(|s| (s.entities.len(), s.value_count, s.unique_values.len()))
.unwrap_or((0, 0, 0));
let (entity_count, value_count, unique_values) = stats.map_or((0, 0, 0), |s| {
(s.entities.len(), s.value_count, s.unique_values.len())
});
// Check if there's an existing (builtin) schema
let is_builtin = self
.schema_registry
.get(&schema.id)
.map(|s| s.builtin)
.unwrap_or(false);
.is_some_and(|s| s.builtin);
SchemaSummaryEntry {
predicate: schema.id.clone(),
@@ -578,7 +578,7 @@ impl Memvid {
/// * `engine` - The enrichment engine to run
///
/// # Returns
/// A tuple of (frames_processed, cards_extracted).
/// A tuple of (`frames_processed`, `cards_extracted`).
pub fn run_enrichment(
&mut self,
engine: &dyn crate::enrich::EnrichmentEngine,
@@ -591,7 +591,11 @@ impl Memvid {
for frame_id in unenriched {
// Get frame data
let Some(frame) = self.toc.frames.get(frame_id as usize) else {
// Safe frame lookup
let Ok(index) = usize::try_from(frame_id) else {
continue;
};
let Some(frame) = self.toc.frames.get(index) else {
continue;
};
let frame = frame.clone();
@@ -628,10 +632,10 @@ impl Memvid {
let card_count = cards.len();
// Store cards
let card_ids = if !cards.is_empty() {
self.put_memory_cards(cards)?
} else {
let card_ids = if cards.is_empty() {
Vec::new()
} else {
self.put_memory_cards(cards)?
};
// Record enrichment
+10 -1
View File
@@ -63,7 +63,7 @@ impl Memvid {
/// Add a mesh edge (relationship) to the Logic-Mesh.
///
/// The edge is deduplicated by (from, to, link_type).
/// The edge is deduplicated by (from, to, `link_type`).
///
/// # Arguments
/// * `edge` - The mesh edge to add
@@ -95,6 +95,7 @@ impl Memvid {
///
/// # Returns
/// A list of entities found by traversing the relationships.
#[must_use]
pub fn follow(&self, start: &str, link: &str, hops: usize) -> Vec<FollowResult> {
self.logic_mesh.follow(start, link, hops)
}
@@ -106,6 +107,7 @@ impl Memvid {
///
/// # Returns
/// The matching node if found.
#[must_use]
pub fn find_entity(&self, name: &str) -> Option<&MeshNode> {
self.logic_mesh.find_node(name)
}
@@ -117,6 +119,7 @@ impl Memvid {
///
/// # Returns
/// A list of entity nodes that have mentions in the specified frame.
#[must_use]
pub fn frame_entities(&self, frame_id: FrameId) -> Vec<&MeshNode> {
self.logic_mesh
.nodes
@@ -132,6 +135,7 @@ impl Memvid {
///
/// # Returns
/// A list of entity nodes matching the specified kind.
#[must_use]
pub fn entities_by_kind(&self, kind: EntityKind) -> Vec<&MeshNode> {
self.logic_mesh
.nodes
@@ -144,6 +148,7 @@ impl Memvid {
///
/// # Returns
/// Statistics including node count, edge count, and breakdowns by kind/link type.
#[must_use]
pub fn logic_mesh_stats(&self) -> LogicMeshStats {
self.logic_mesh.stats()
}
@@ -152,16 +157,19 @@ impl Memvid {
///
/// # Returns
/// `true` if the mesh has nodes or edges.
#[must_use]
pub fn has_logic_mesh(&self) -> bool {
!self.logic_mesh.is_empty()
}
/// Get the number of entity nodes in the mesh.
#[must_use]
pub fn mesh_node_count(&self) -> usize {
self.logic_mesh.nodes.len()
}
/// Get the number of relationship edges in the mesh.
#[must_use]
pub fn mesh_edge_count(&self) -> usize {
self.logic_mesh.edges.len()
}
@@ -169,6 +177,7 @@ impl Memvid {
/// Get entities for a frame as `SearchHitEntity` for search metadata.
///
/// Returns entities from the Logic-Mesh that appear in the given frame.
#[must_use]
pub fn frame_entities_for_search(&self, frame_id: FrameId) -> Vec<SearchHitEntity> {
self.logic_mesh
.nodes
+1
View File
@@ -1,5 +1,6 @@
//! Core `Memvid` type orchestrating `.mv2` lifecycle and mutations.
mod acl;
pub mod ask;
pub mod audit;
#[cfg(feature = "parallel_segments")]
+576 -292
View File
File diff suppressed because it is too large Load Diff
+160 -54
View File
@@ -9,9 +9,9 @@ use tempfile::TempDir;
use crate::memvid::lifecycle::Memvid;
use crate::types::{
AdaptiveConfig, AdaptiveResult, AdaptiveStats, EmbeddingQualityStats, Frame, FrameId,
FrameStatus, SearchHit, TimelineEntry, TimelineQuery, VecSegmentDescriptor,
compute_embedding_quality, find_adaptive_cutoff,
AclContext, AclEnforcementMode, AdaptiveConfig, AdaptiveResult, AdaptiveStats,
EmbeddingQualityStats, Frame, FrameId, FrameStatus, SearchHit, TimelineEntry, TimelineQuery,
VecSegmentDescriptor, compute_embedding_quality, find_adaptive_cutoff,
};
use crate::{LexSearchHit, MemvidError, Result, VecSearchHit};
@@ -19,10 +19,17 @@ impl Memvid {
pub fn enable_lex(&mut self) -> Result<()> {
self.ensure_writable()?;
if self.lex_enabled {
// If index exists on disk but not in memory, load it
#[cfg(feature = "lex")]
if self.lex_index.is_none() && crate::memvid::lifecycle::has_lex_index(&self.toc) {
self.load_lex_index_from_manifest()?;
{
// If index exists on disk but not in memory, load it
if self.lex_index.is_none() && crate::memvid::lifecycle::has_lex_index(&self.toc) {
self.load_lex_index_from_manifest()?;
}
// Ensure Tantivy engine is running even if lex was already enabled
// (e.g. create() set lex_enabled=true but tantivy may have been lost)
if self.tantivy.is_none() {
self.init_tantivy()?;
}
}
return Ok(());
}
@@ -87,6 +94,7 @@ impl Memvid {
bytes_length: 0,
checksum: empty_checksum,
compression_mode: self.vec_compression.clone(),
model: self.vec_model.clone(),
});
}
@@ -94,23 +102,53 @@ impl Memvid {
Ok(())
}
/// Set the expected embedding model for the vector index.
///
/// If the index is already bound to a model (from a previous session or call),
/// this validates that the requested model matches the existing one.
/// If unbound, it binds the index to the new model.
pub fn set_vec_model(&mut self, model: &str) -> Result<()> {
if let Some(existing) = &self.vec_model {
if existing != model {
return Err(MemvidError::ModelMismatch {
expected: existing.clone(),
actual: model.to_string(),
});
}
} else {
self.vec_model = Some(model.to_string());
// If manifest exists, update it to persist the binding
if let Some(manifest) = self.toc.indexes.vec.as_mut() {
manifest.model = Some(model.to_string());
self.dirty = true;
}
}
Ok(())
}
pub fn search_vec(&mut self, query: &[f32], limit: usize) -> Result<Vec<VecSearchHit>> {
if !self.vec_enabled {
return Err(MemvidError::VecNotEnabled);
}
let mut ensured_vec_index = false;
let expected_dim = match self.effective_vec_index_dimension()? {
Some(dim) => dim,
None => {
self.ensure_vec_index()?;
ensured_vec_index = true;
self.vec_index
.as_ref()
.and_then(|index| index.entries().next().map(|(_, emb)| emb.len() as u32))
.unwrap_or(0)
}
let expected_dim = if let Some(dim) = self.effective_vec_index_dimension()? {
dim
} else {
self.ensure_vec_index()?;
ensured_vec_index = true;
self.vec_index
.as_ref()
.and_then(|index| {
index
.entries()
.next()
.map(|(_, emb)| u32::try_from(emb.len()).unwrap_or(0))
})
.unwrap_or(0)
};
if expected_dim > 0 && query.len() as u32 != expected_dim {
// Safe: embedding dimensions are small (< few thousands)
#[allow(clippy::cast_possible_truncation)]
if expected_dim > 0 && (query.len() as u32) != expected_dim {
return Err(MemvidError::VecDimensionMismatch {
expected: expected_dim,
actual: query.len(),
@@ -235,6 +273,28 @@ impl Memvid {
top_k: usize,
snippet_chars: usize,
scope: Option<&str>,
) -> Result<crate::types::SearchResponse> {
self.vec_search_with_embedding_acl(
query,
query_embedding,
top_k,
snippet_chars,
scope,
None,
AclEnforcementMode::Audit,
)
}
/// Perform pure vector search using a pre-computed query embedding with ACL filtering.
pub fn vec_search_with_embedding_acl(
&mut self,
query: &str,
query_embedding: &[f32],
top_k: usize,
snippet_chars: usize,
scope: Option<&str>,
acl_context: Option<&AclContext>,
acl_enforcement_mode: AclEnforcementMode,
) -> Result<crate::types::SearchResponse> {
use super::helpers::{build_context, timestamp_to_rfc3339};
use crate::types::{
@@ -249,18 +309,24 @@ impl Memvid {
// Validate embedding dimension BEFORE searching to prevent silent wrong results.
// For segment-only memories, dimension may only be discoverable after loading segments.
let mut ensured_vec_index = false;
let expected_dim = match self.effective_vec_index_dimension()? {
Some(dim) => dim,
None => {
self.ensure_vec_index()?;
ensured_vec_index = true;
self.vec_index
.as_ref()
.and_then(|index| index.entries().next().map(|(_, emb)| emb.len() as u32))
.unwrap_or(0)
}
let expected_dim = if let Some(dim) = self.effective_vec_index_dimension()? {
dim
} else {
self.ensure_vec_index()?;
ensured_vec_index = true;
self.vec_index
.as_ref()
.and_then(|index| {
index
.entries()
.next()
.map(|(_, emb)| u32::try_from(emb.len()).unwrap_or(0))
})
.unwrap_or(0)
};
if expected_dim > 0 && query_embedding.len() as u32 != expected_dim {
// Safe: embedding dimensions are small
#[allow(clippy::cast_possible_truncation)]
if expected_dim > 0 && (query_embedding.len() as u32) != expected_dim {
return Err(MemvidError::VecDimensionMismatch {
expected: expected_dim,
actual: query_embedding.len(),
@@ -294,6 +360,7 @@ impl Memvid {
context: build_context(&[]),
next_cursor: None,
engine: SearchEngineKind::Hybrid,
stale_index_skips: 0,
});
}
@@ -303,7 +370,14 @@ impl Memvid {
for vec_hit in vec_hits {
// Apply scope filter if provided
let frame = match self.toc.frames.get(vec_hit.frame_id as usize) {
// Apply scope filter if provided
let frame_idx = if let Ok(idx) = usize::try_from(vec_hit.frame_id) {
idx
} else {
continue;
};
let frame = match self.toc.frames.get(frame_idx) {
Some(f) => f.clone(),
None => continue,
};
@@ -346,6 +420,7 @@ impl Memvid {
created_at: timestamp_to_rfc3339(frame.timestamp),
content_dates: frame.content_dates.clone(),
entities: Vec::new(),
extra_metadata: frame.extra_metadata.clone(),
#[cfg(feature = "temporal_track")]
temporal: None,
};
@@ -374,6 +449,7 @@ impl Memvid {
#[cfg(feature = "temporal_track")]
super::helpers::attach_temporal_metadata(self, &mut hits)?;
self.apply_acl_to_search_hits(&mut hits, acl_context, acl_enforcement_mode)?;
let context = build_context(&hits);
Ok(SearchResponse {
@@ -389,6 +465,7 @@ impl Memvid {
context,
next_cursor: None,
engine: SearchEngineKind::Hybrid,
stale_index_skips: 0,
})
}
@@ -419,17 +496,41 @@ impl Memvid {
config: AdaptiveConfig,
snippet_chars: usize,
scope: Option<&str>,
) -> Result<AdaptiveResult<SearchHit>> {
self.search_adaptive_acl(
query,
query_embedding,
config,
snippet_chars,
scope,
None,
AclEnforcementMode::Audit,
)
}
/// Perform adaptive vector search with ACL filtering.
pub fn search_adaptive_acl(
&mut self,
query: &str,
query_embedding: &[f32],
config: AdaptiveConfig,
snippet_chars: usize,
scope: Option<&str>,
acl_context: Option<&AclContext>,
acl_enforcement_mode: AclEnforcementMode,
) -> Result<AdaptiveResult<SearchHit>> {
use std::time::Instant;
if !config.enabled {
// Fall back to standard search with max_results as top_k
let response = self.vec_search_with_embedding(
let response = self.vec_search_with_embedding_acl(
query,
query_embedding,
config.max_results,
snippet_chars,
scope,
acl_context,
acl_enforcement_mode,
)?;
return Ok(AdaptiveResult {
results: response.hits,
@@ -448,12 +549,14 @@ impl Memvid {
let start_time = Instant::now();
// Over-retrieve: get max_results to have enough candidates
let response = self.vec_search_with_embedding(
let response = self.vec_search_with_embedding_acl(
query,
query_embedding,
config.max_results,
snippet_chars,
scope,
acl_context,
acl_enforcement_mode,
)?;
if response.hits.is_empty() {
@@ -557,7 +660,7 @@ impl Memvid {
Ok(Some(compute_embedding_quality(&embeddings)))
}
/// Get frame IDs filtered by Replay parameters (as_of_frame or as_of_ts).
/// Get frame IDs filtered by Replay parameters (`as_of_frame` or `as_of_ts`).
/// Used for time-travel memory views.
pub(crate) fn get_replay_frame_ids(
&self,
@@ -624,7 +727,7 @@ impl Memvid {
#[allow(dead_code)]
fn materialize_tantivy_segments(&mut self, segments: &[EmbeddedLexSegment]) -> Result<TempDir> {
let dir = TempDir::new().map_err(|err| MemvidError::Tantivy {
reason: format!("failed to allocate Tantivy work directory: {}", err),
reason: format!("failed to allocate Tantivy work directory: {err}"),
})?;
if segments.is_empty() {
return Ok(dir);
@@ -635,11 +738,11 @@ impl Memvid {
.metadata()
.map(|meta| meta.len())
.map_err(|err| MemvidError::Tantivy {
reason: format!("failed to inspect memvid file metadata: {}", err),
reason: format!("failed to inspect memvid file metadata: {err}"),
})?;
let mut data_limit = self.header.footer_offset;
let mut buffer = vec![0u8; 64 * 1024];
let cursor = self.file.seek(SeekFrom::Current(0))?;
let cursor = self.file.stream_position()?;
for segment in segments {
let dest = dir.path().join(&segment.path);
if let Some(parent) = dest.parent() {
@@ -674,7 +777,7 @@ impl Memvid {
if self.align_footer_with_catalog()? {
file_len = self.file.metadata().map(|meta| meta.len()).map_err(|err| {
MemvidError::Tantivy {
reason: format!("failed to refresh memvid file metadata: {}", err),
reason: format!("failed to refresh memvid file metadata: {err}"),
}
})?;
data_limit = self.header.footer_offset;
@@ -695,6 +798,8 @@ impl Memvid {
self.file.seek(SeekFrom::Start(segment.bytes_offset))?;
let mut remaining = segment.bytes_length;
while remaining > 0 {
// Safe: chunk is at most buffer.len() which is usize
#[allow(clippy::cast_possible_truncation)]
let chunk = remaining.min(buffer.len() as u64) as usize;
if let Err(err) = self.file.read_exact(&mut buffer[..chunk]) {
return Err(MemvidError::Tantivy {
@@ -719,7 +824,18 @@ impl Memvid {
return Ok(());
}
let segments = if !self.toc.segment_catalog.tantivy_segments.is_empty() {
let segments = if self.toc.segment_catalog.tantivy_segments.is_empty() {
match self.lex_storage.read() {
Ok(storage) => {
if storage.is_empty() {
None
} else {
Some(storage.segments().cloned().collect::<Vec<_>>())
}
}
Err(_) => None,
}
} else {
Some(
self.toc
.segment_catalog
@@ -733,17 +849,6 @@ impl Memvid {
})
.collect::<Vec<_>>(),
)
} else {
match self.lex_storage.read() {
Ok(storage) => {
if storage.is_empty() {
None
} else {
Some(storage.segments().cloned().collect::<Vec<_>>())
}
}
Err(_) => None,
}
};
let mut engine = match segments {
@@ -780,9 +885,7 @@ impl Memvid {
// Trust existing Tantivy segments, don't rebuild
false
} else {
expected_docs
.map(|expected| expected != actual_docs)
.unwrap_or(true)
expected_docs != Some(actual_docs)
};
if needs_rebuild {
@@ -807,6 +910,7 @@ impl Memvid {
Ok(())
}
#[must_use]
pub fn vec_segment_descriptor(&self, segment_id: u64) -> Option<VecSegmentDescriptor> {
self.toc
.segment_catalog
@@ -832,10 +936,11 @@ impl Memvid {
}
/// Default maximum payload size for text indexing (256 MiB)
/// Can be overridden via MEMVID_MAX_INDEX_PAYLOAD environment variable
/// Can be overridden via `MEMVID_MAX_INDEX_PAYLOAD` environment variable
pub const DEFAULT_MAX_INDEX_PAYLOAD: u64 = 256 * 1024 * 1024;
/// Get the maximum indexable payload size from environment or use default
#[must_use]
pub fn max_index_payload() -> u64 {
std::env::var("MEMVID_MAX_INDEX_PAYLOAD")
.ok()
@@ -844,6 +949,7 @@ pub fn max_index_payload() -> u64 {
}
/// Check if a MIME type represents text-based content that should be indexed
#[must_use]
pub fn is_text_indexable_mime(mime: &str) -> bool {
let mime_lower = mime.to_lowercase();
@@ -895,6 +1001,7 @@ pub fn is_text_indexable_mime(mime: &str) -> bool {
}
/// Check if a frame should be indexed for text search
#[must_use]
pub fn is_frame_text_indexable(frame: &crate::types::Frame) -> bool {
// Must be active
if frame.status != crate::types::FrameStatus::Active {
@@ -923,8 +1030,7 @@ pub fn is_frame_text_indexable(frame: &crate::types::Frame) -> bool {
frame
.search_text
.as_ref()
.map(|t| !t.trim().is_empty())
.unwrap_or(false)
.is_some_and(|t| !t.trim().is_empty())
}
#[cfg(feature = "lex")]
+22 -16
View File
@@ -96,14 +96,14 @@ impl Memvid {
return Ok(());
}
let bytes = match self.read_range(manifest.bytes_offset, manifest.bytes_length) {
Ok(bytes) => bytes,
Err(_) => {
let bytes =
if let Ok(bytes) = self.read_range(manifest.bytes_offset, manifest.bytes_length) {
bytes
} else {
// Don't disable lex if loading fails - keep it enabled
self.lex_index = None;
return Ok(());
}
};
};
match LexIndex::decode(&bytes) {
Ok(mut index) => {
self.hydrate_lex_index_metadata(&mut index);
@@ -123,6 +123,9 @@ impl Memvid {
}
pub(crate) fn load_vec_index_from_manifest(&mut self) -> Result<()> {
// Load the model name from the manifest regardless of validation success
self.vec_model = self.toc.indexes.vec.as_ref().and_then(|m| m.model.clone());
if let Some(manifest) = &self.toc.indexes.vec {
// Empty manifest (placeholder for enabled but not yet populated index)
if manifest.bytes_length == 0 {
@@ -130,15 +133,15 @@ impl Memvid {
return Ok(());
}
let bytes = match self.read_range(manifest.bytes_offset, manifest.bytes_length) {
Ok(bytes) => bytes,
Err(_) => {
let bytes =
if let Ok(bytes) = self.read_range(manifest.bytes_offset, manifest.bytes_length) {
bytes
} else {
self.vec_index = None;
// Don't disable vec if loading fails - keep it enabled
// self.vec_enabled = false;
return Ok(());
}
};
};
match catch_unwind(AssertUnwindSafe(|| VecIndex::decode(&bytes))) {
Ok(Ok(index)) => self.vec_index = Some(index),
Ok(Err(_)) | Err(_) => {
@@ -166,13 +169,13 @@ impl Memvid {
return Ok(());
}
let bytes = match self.read_range(manifest.bytes_offset, manifest.bytes_length) {
Ok(bytes) => bytes,
Err(_) => {
let bytes =
if let Ok(bytes) = self.read_range(manifest.bytes_offset, manifest.bytes_length) {
bytes
} else {
self.clip_index = None;
return Ok(());
}
};
};
match catch_unwind(AssertUnwindSafe(|| ClipIndex::decode(&bytes))) {
Ok(Ok(index)) => self.clip_index = Some(index),
Ok(Err(_)) | Err(_) => {
@@ -196,6 +199,8 @@ impl Memvid {
});
}
self.file.seek(SeekFrom::Start(offset))?;
// Safe: length is checked against MAX_INDEX_BYTES above
#[allow(clippy::cast_possible_truncation)]
let mut buf = vec![0u8; length as usize];
self.file.read_exact(&mut buf)?;
Ok(buf)
@@ -273,7 +278,8 @@ impl Memvid {
fn hydrate_lex_index_metadata(&self, index: &mut LexIndex) {
for document in index.documents_mut() {
let frame_meta = self.toc.frames.get(document.frame_id as usize);
let frame_idx = usize::try_from(document.frame_id).ok();
let frame_meta = frame_idx.and_then(|idx| self.toc.frames.get(idx));
if document.uri.is_none() {
let derived = frame_meta
+35 -14
View File
@@ -39,17 +39,24 @@ pub(super) fn search_with_lex_fallback(
let max_snippets_per_doc = request.top_k.max(1);
let mut evaluated = Vec::new();
let mut stale_skips = 0u32;
for matched in &matches {
if let Some(filter) = candidate_filter {
if !filter.contains(&matched.frame_id) {
continue;
}
}
let frame_meta = memvid.toc.frames.get(matched.frame_id as usize).ok_or(
MemvidError::InvalidTimeIndex {
reason: "frame id out of range".into(),
},
)?;
let Some(frame_meta) = usize::try_from(matched.frame_id)
.ok()
.and_then(|idx| memvid.toc.frames.get(idx))
else {
tracing::warn!(
frame_id = matched.frame_id,
"skipping search hit with stale frame_id"
);
stale_skips = stale_skips.saturating_add(1);
continue;
};
let content_lower = matched.content.to_ascii_lowercase();
let ctx = EvaluationContext {
frame: frame_meta,
@@ -85,19 +92,28 @@ pub(super) fn search_with_lex_fallback(
let mut hits = Vec::new();
let mut produced = 0usize;
for (matched, slices) in evaluated {
let frame_meta = memvid
let Some(frame_meta) = memvid
.toc
.frames
.get(matched.frame_id as usize)
.get(usize::try_from(matched.frame_id).unwrap_or(usize::MAX))
.cloned()
.ok_or(MemvidError::InvalidTimeIndex {
reason: "frame id out of range".into(),
})?;
else {
tracing::warn!(
frame_id = matched.frame_id,
"skipping stale frame_id in snippet assembly"
);
continue;
};
let canonical = memvid.frame_content(&frame_meta)?;
let canonical_limit = frame_meta
.canonical_length
.map(|len| len as usize)
.unwrap_or_else(|| canonical.len());
let canonical_limit = frame_meta.canonical_length.map_or_else(
|| canonical.len(),
|len| {
// Safe: canonical length is reasonably small string length
#[allow(clippy::cast_possible_truncation)]
let l = len as usize;
l
},
);
let canonical_len = canonical.len();
let effective_len = canonical_limit.min(canonical_len);
let uri = matched
@@ -134,6 +150,7 @@ pub(super) fn search_with_lex_fallback(
created_at: timestamp_to_rfc3339(frame_meta.timestamp),
content_dates: frame_meta.content_dates.clone(),
entities: Vec::new(),
extra_metadata: frame_meta.extra_metadata.clone(),
#[cfg(feature = "temporal_track")]
temporal: None,
};
@@ -193,6 +210,7 @@ pub(super) fn search_with_lex_fallback(
context,
next_cursor,
engine: SearchEngineKind::LexFallback,
stale_index_skips: stale_skips,
})
}
@@ -243,6 +261,7 @@ pub(super) fn search_with_filters_only(
context: build_context(&[]),
next_cursor: None,
engine: SearchEngineKind::LexFallback,
stale_index_skips: 0,
});
}
@@ -265,6 +284,7 @@ pub(super) fn search_with_filters_only(
created_at: timestamp_to_rfc3339(frame.timestamp),
content_dates: frame.content_dates.clone(),
entities: Vec::new(),
extra_metadata: frame.extra_metadata.clone(),
#[cfg(feature = "temporal_track")]
temporal: None,
};
@@ -314,5 +334,6 @@ pub(super) fn search_with_filters_only(
context,
next_cursor,
engine: SearchEngineKind::LexFallback,
stale_index_skips: 0,
})
}
+22 -11
View File
@@ -1,3 +1,5 @@
// Safe unwrap: guaranteed non-empty vector operations.
#![allow(clippy::unwrap_used)]
use crate::MemvidError;
use crate::Result;
use crate::memvid::lifecycle::Memvid;
@@ -31,6 +33,7 @@ pub(super) fn empty_search_response(
context: String::new(),
next_cursor: None,
engine,
stale_index_skips: 0,
}
}
@@ -67,7 +70,7 @@ pub(super) fn parse_cursor(cursor: Option<&str>, total_hits: usize) -> Result<us
/// Build context for LLM from search hits using a multi-document strategy.
///
/// Key design decisions for deterministic, comprehensive context:
/// 1. Uses BTreeMap for deterministic iteration order (sorted by URI)
/// 1. Uses `BTreeMap` for deterministic iteration order (sorted by URI)
/// 2. Includes top hits from MULTIPLE documents for diverse context
/// 3. Prioritizes by rank while ensuring document diversity
/// 4. Maximum 24 hits for balanced context (not too much noise, not too little coverage)
@@ -89,7 +92,7 @@ pub(crate) fn build_context(hits: &[SearchHit]) -> String {
.next()
.unwrap_or(&hit.uri)
.to_ascii_lowercase();
let entry = groups.entry(base).or_insert_with(GroupSummary::default);
let entry = groups.entry(base).or_default();
entry.indices.push(idx);
entry.total_matches += hit.matches.max(1);
entry.best_rank = entry.best_rank.min(hit.rank);
@@ -342,13 +345,18 @@ pub(crate) fn attach_temporal_metadata(memvid: &mut Memvid, hits: &mut [SearchHi
let text = if mention_end > mention_start {
if !canonical_cache.contains_key(&frame_id) {
let frame = memvid.toc.frames.get(frame_id as usize).cloned().ok_or(
MemvidError::InvalidTimeIndex {
reason: "frame id out of range".into(),
},
)?;
let content = memvid.frame_content(&frame)?;
canonical_cache.insert(frame_id, content);
match memvid.toc.frames.get(frame_id as usize).cloned() {
Some(frame) => {
let content = memvid.frame_content(&frame)?;
canonical_cache.insert(frame_id, content);
}
None => {
tracing::warn!(
frame_id,
"skipping temporal text for stale frame_id"
);
}
}
}
canonical_cache.get(&frame_id).and_then(|content| {
if mention_end <= content.len() {
@@ -396,7 +404,7 @@ pub(crate) fn attach_temporal_metadata(memvid: &mut Memvid, hits: &mut [SearchHi
/// Enrich search hits with entities from the Logic-Mesh.
///
/// For each hit, looks up entities that are associated with the hit's frame.
/// If the frame is a DocumentChunk (page), also checks the parent document frame
/// If the frame is a `DocumentChunk` (page), also checks the parent document frame
/// for entities since NER extraction happens on the full document.
pub(super) fn enrich_hits_with_entities(hits: &mut [SearchHit], memvid: &Memvid) {
for hit in hits.iter_mut() {
@@ -404,7 +412,10 @@ pub(super) fn enrich_hits_with_entities(hits: &mut [SearchHit], memvid: &Memvid)
// If no entities found and this is a chunk, check the parent frame
if entities.is_empty() {
if let Some(frame) = memvid.toc.frames.get(hit.frame_id as usize) {
if let Some(frame) = usize::try_from(hit.frame_id)
.ok()
.and_then(|idx| memvid.toc.frames.get(idx))
{
if let Some(parent_id) = frame.parent_id {
entities = memvid.frame_entities_for_search(parent_id);
}
+19 -2
View File
@@ -31,7 +31,7 @@ pub use api::{
#[cfg(feature = "lex")]
use fallback::{search_with_filters_only, search_with_lex_fallback};
use helpers::empty_search_response;
use helpers::{build_context, empty_search_response};
#[cfg(feature = "lex")]
pub use tantivy::parse_content_date_to_timestamp;
#[cfg(feature = "lex")]
@@ -48,6 +48,13 @@ impl Memvid {
return Err(MemvidError::LexNotEnabled);
}
// Lazy-init Tantivy if lex is enabled but engine is missing.
// This can happen when a wrapper re-enables lex on an already-enabled
// instance, or after a staging-lock rollback lost the engine reference.
if self.tantivy.is_none() {
self.init_tantivy()?;
}
let start_time = Instant::now();
// parse_query can return structured tokens; we only keep non-empty, lower-cased terms.
let parsed = crate::search::parse_query(&request.query)?;
@@ -55,7 +62,7 @@ impl Memvid {
query_tokens.retain(|token| !token.trim().is_empty());
query_tokens = query_tokens
.into_iter()
.map(|token| token.to_ascii_lowercase())
.map(|s| s.as_str().to_ascii_lowercase())
.collect::<BTreeSet<_>>()
.into_iter()
.collect();
@@ -256,6 +263,16 @@ impl Memvid {
}
};
self.apply_acl_to_search_hits(
&mut response.hits,
request.acl_context.as_ref(),
request.acl_enforcement_mode,
)?;
if request.acl_enforcement_mode == crate::types::AclEnforcementMode::Enforce {
response.total_hits = response.hits.len();
response.context = build_context(&response.hits);
}
// Enrich hits with Logic-Mesh entities if mesh is available
if self.has_logic_mesh() {
helpers::enrich_hits_with_entities(&mut response.hits, self);

Some files were not shown because too many files have changed in this diff Show More