19 Commits

Author SHA1 Message Date
Jalin Wang 1afdea8dc5 feat(python): expose group-by search to Python API (#561)
Co-authored-by: jiliang.ljl <jiliang.ljl@alibaba-inc.com>
2026-07-13 20:22:07 +08:00
egolearner a90ec5b1d6 fix(fts): preserve zero-match filter semantics (#584) 2026-07-10 15:20:51 +08:00
egolearner df6af2f05d feat: Implement standard tokenizer based on UAX29 (#547) 2026-07-09 10:01:56 +08:00
egolearner aad5718112 feat(query): add VectorViewClause zero-copy path and unify validate (#478)
* feat(query): add VectorViewClause zero-copy path and unify validate

- Add VectorViewClause (string_view-based) as zero-copy counterpart to
  VectorClause; variant now holds VectorClause | VectorViewClause | FtsClause
- Add QueryTarget::get_vector_view() unified accessor via std::visit,
  returns optional<VectorViewClause> regardless of which variant is held
- Split validate_and_sanitize into QueryTarget::validate (read-only) +
  sanitize_sparse_vector (mutate); validate handles both VectorClause and
  VectorViewClause via get_vector_view()
- Collection::Query passes original request directly to sqlengine when no
  sparse sanitization is needed; only copies when sort is required
- Change build_query_info/BuildSQLInfoFromSearchQuery to take const
  SearchQuery& so VectorMatrixNode string_views point to caller's data
2026-06-22 11:32:26 +08:00
feihongxu0824 4679a8f39d feat: support zvec core-only build (#481) 2026-06-09 23:45:42 +08:00
egolearner 443500dc45 feat: add FTS support for Collection::CreateIndex/DropIndex (#445)
* feat: add FTS support for Collection::CreateIndex/DropIndex

Enable dynamic creation and removal of FTS indexes on existing STRING
columns through the standard CreateIndex/DropIndex API, matching the
lifecycle model already used by vector and scalar (invert) indexes.

Key changes:
- New FtsIndexer class (fts_indexer.h/cc) encapsulating per-segment FTS
  RocksDB management: multi-field lifecycle, snapshot, insert, seal
- New BlockType::FTS_INDEX with block_id-based directory naming
  (fts.<block_id>.rocksdb) for crash-safe snapshot-and-swap
- Segment::create_fts_index builds FTS index on a snapshot copy by
  scanning forward store, then outputs new SegmentMeta + FtsIndexer
  for atomic reload (same pattern as create_scalar_index)
- Segment::drop_fts_index snapshots, removes field CFs, outputs updated
  meta (or nullptr when last FTS field is removed)
- Collection layer wires FTS into the existing task dispatch, version
  update, and reload loops alongside vector/invert paths
- CreateIndex/DropIndex reject unsupported index types explicitly
  instead of falling through to the wrong branch

* fixup! feat: add FTS support for Collection::CreateIndex/DropIndex

    fix: address review comments for FTS CreateIndex/DropIndex

    - Reject CreateIndex when column already has a different index type
      (e.g. FTS on an INVERT-indexed column) at both Collection and
      Segment layers
    - Allow same-type different-params CreateIndex to rebuild the index
      (remove old + create new + replay data), aligned with INVERT behavior
    - Return OK when CreateIndex is called with identical params
    - Rename operator[] to get() in both FtsIndexer and InvertedIndexer
    - Add test cases: create→drop→create→drop cycle, and params-change
      rebuild with case-sensitivity verification

* android skip Feature_CreateOrDropFtsIndex

* fixup! feat: add FTS support for Collection::CreateIndex/DropIndex
2026-06-03 17:56:56 +08:00
Qinren Zhou dbea635019 refactor: clarify segment-local row ID handling and tests and fixes bugs (#432) 2026-06-03 15:20:45 +08:00
Qinren Zhou 8e8bb81db0 refactor: tidy query APIs and execution (#431) 2026-06-01 21:07:35 +08:00
egolearner 02bfb31cf5 feat: add fts support (#408)
Add BM25-based full-text search with CJK (jieba) tokenization, supporting
  query_string and match_string syntax, phrase queries, boolean operators
  (AND/OR/NOT/MUST), and hybrid retrieval with existing vector search.

  ## Core
  - BitPacked posting format with block-max WAND pruning
  - Tokenizer pipeline: jieba (cut/cut_for_search/hmm/full), whitespace,
    lowercase, with extensible pipeline composition
  - Query parser: boolean operators, phrase queries, field scoping,
    boost, MUST(+) modifier inside OR (ES query_string semantics)
  - AST rewriter: dedup repeated terms with linear boost aggregation,
    flatten same-type composites, canonicalize OR-with-must_not into AND
    wrapper, empty-node propagation, contradiction detection
  - FTS reduce/merge integrated into Optimize compaction
  - Multi-segment score-descending sort
  - Auto-register bundled jieba dict on SDK import

  ## Performance
  - Block-max WAND with cached block_max_info_for (single binary search)
  - AVX2/SSE bitpacked encoding with cross-arch scalar fallback
  - MultiGet for batch posting retrieval and phrase position verification
  - HashSkipList memtable for posting writes
  - PinnableSlice zero-copy reads
  - Filter pushdown into composite iterators (Disjunction/Conjunction/Phrase)
  - Candidate-driven (brute-force) evaluation for selective invert filters
  - Precomputed BM25 IDF weights, cached SIMD dispatch pointers
  - Shortest-list anchor for phrase position matching
  - Single-open per-term posting iterator

  ## Query
  - Tokenize query terms through the same pipeline as indexing
  - EmptyNode for zero-token queries (all stop-words / punctuation)
  - Backslash unescape after lexing in query parser
  - Schema allows collections without vector fields (FTS-only use case)
  - Create/Drop Index validates supported index types
  - FTS fields disallowed in SQL filter expressions

  ## Bindings
  - C API: fts query params, brute-force ratio config
  - Python SDK: FTS search, jieba dict auto-registration

  ## Internals
  - Bypass cppjieba::Jieba to drop KeywordExtractor (~12MB fewer required files)
  - Hide tokenizer pipeline from public header (Pimpl-style FtsState)
  - ListColumnFamilies to avoid double-open on segment load
  - Reorganized fts_column into tokenizer/, posting/, iterator/ subdirs
2026-06-01 15:02:54 +08:00
egolearner 8dcb6cbd7f refactor: drop VectorQuery, unify single-target query on SearchQuery (#428) 2026-05-29 16:36:09 +08:00
lichen2015 e0ba23179b feat: fetch() add output_fields param (#358) 2026-05-27 22:58:49 +08:00
kgeg401 1e25294c07 feat(ci): integrate clang-tidy for changed C/C++ files (#116)
* feat(ci): add clang-tidy checks for changed cpp files

* fix(ci): correct clang-tidy workflow heredoc indentation

* test: stabilize fp16 euclidean matrix comparison

* test: fix fp16 matrix CI and clang-tidy warnings

* ci: scope clang-tidy PR and filter compile-db files

* check nullptr only

* fix nullptr

* fix: format

* chore: ignore c

* fix: tests files

* fix: workflow

* fix: specify clang-tidy version

* fix: filter regex

* fix: more files

* fix: update version

* add thirdparty cache

* add pipeline dependency

* fix clang-tidy trigger

---------

Co-authored-by: kgeg401 <kgeg401@users.noreply.github.com>
Co-authored-by: jiliang.ljl <jiliang.ljl@alibaba-inc.com>
2026-04-20 20:14:21 +08:00
egolearner e5ba11b6fe feat: add hnsw-rabitq support (#69)
* feat: add hnsw-rabitq

* undefine transform

* support config rotator type

* support sample_count

* fix ut

* refactor: update interface

* refactor: update interface

* update rabitq index params

* fix interface update

* fix searcher test

* fix streamer test

* streamer support bf and add more ut

* rm env check

* add collection ut

* add schema check

* add rabitq query param binding

* add integration test

* fix local_builder

* cleanup dist calculator

* add RaBitQ-Library submodule

* cleanup rabitq converter/reformer

* add files

* disable build on mac

* disable Feature_Optimize_HNSW_RABITQ

* disable python/tests/test_collection_hnsw_rabitq.py:13

* check avx2/avx512

* fix refine

* fix mac ci

* add dimension check

* fix mac ci

* fix ci

* add missing lib

* fix centroids selection for Cosine/InnerProduct

* rename hnsw-rabitq to hnsw_rabitq

* address comments

* fix compile

* fix rabitqlib name

* fix mac compile

* check avx2/avx512 support

* runtime check again compile-time

* check AUTO_DETECT_ARCH

* rm debug log

* fix

* rabitq use avx2 by default

* add missing file

* fix search_bf/group_by dist

* address comments

* fix typo

* address comments

* rabitq support disable id_map

* address comments
2026-03-19 17:21:42 +08:00
Qinren Zhou 9314cb14a6 fix: use per-block filter instead of per-segment filter during query (#223)
* fix: id mismatch

* fix: remove debug string

* fix: use protected

* chore(comment): add comments

* feat: more ut

* Update vector_recall_test.cc

* fix: resolve comments
2026-03-16 10:55:51 +08:00
feihongxu0824 fb0b900a80 fix(build): replace CMAKE_SOURCE_DIR with PROJECT_ROOT_DIR for subproject builds (#195) 2026-03-04 20:01:52 +08:00
lichen2015 f1349cc91a fix: remove unnecessary column_name param from the AddColumn API (#59) 2026-02-04 14:44:13 +08:00
feihongxu0824 0ebf5621a6 feat: support core cpp sdk (#30)
* support core cpp sdk

* fix

* fix

* fix

* fix cr
2026-01-26 20:01:44 +08:00
feihongxu0824 e957442355 feat: refact cpp sdk (#27)
* refact cpp sdk

---------

Co-authored-by: zhouqinren.zqr <zhouqinren.zqr@alibaba-inc.com>
2026-01-21 20:05:42 +08:00
sanyi 6524fabd80 Initial commit 2025-12-30 11:02:17 +08:00