chore: import upstream snapshot with attribution

tests/python_client/.dockerignore

@@ -0,0 +1,14 @@
+node_modules
+npm-debug.log
+Dockerfile*
+docker-compose*
+.dockerignore
+.git
+.gitignore
+.env
+*/bin
+*/obj
+README.md
+LICENSE
+.vscode
+__pycache__

tests/python_client/.gitignore

@@ -0,0 +1,21 @@
+# python files
+.pytest_cache
+**/.pytest_cache
+**/volumes
+**/logs
+**/docker-compose.yml
+.idea
+*.html
+*.hdf5
+*.npy
+.python-version
+__pycache__
+.vscode
+
+test_out/
+*.pyc
+
+db/
+logs/
+
+.coverage

tests/python_client/Dockerfile

@@ -0,0 +1,29 @@
+FROM python:3.12-bullseye
+
+RUN apt-get update && apt-get install -y jq
+
+# Define the ARG variable
+ARG PIP_TRUSTED_HOST=""
+ARG PIP_INDEX_URL=""
+ARG PIP_INDEX=""
+ARG PIP_FIND_LINKS=""
+
+# Set the ENV variable
+ENV PIP_TRUSTED_HOST=${PIP_TRUSTED_HOST}
+ENV PIP_INDEX_URL=${PIP_INDEX_URL}
+ENV PIP_INDEX=${PIP_INDEX}
+ENV PIP_FIND_LINKS=${PIP_FIND_LINKS}
+
+
+WORKDIR /milvus
+
+COPY tests/python_client/requirements.txt tests/python_client/requirements.txt
+
+RUN cd tests/python_client && python3 -m pip install --upgrade setuptools \
+    && python3 -m pip install --upgrade pip \
+    && python3 -m pip install --no-cache-dir  -r requirements.txt --timeout 30 --retries 6
+
+COPY tests/python_client tests/python_client
+COPY tests/restful_client tests/restful_client
+COPY tests/restful_client_v2 tests/restful_client_v2
+COPY tests/scripts tests/scripts

tests/python_client/README.md

@@ -0,0 +1,257 @@
+# Guidelines for Test Framework
+
+This document guides you through the Pytest-based PyMilvus test framework.
+
+> You can find the test code on [GitHub](https://github.com/milvus-io/milvus/tree/master/tests/python_client).
+
+
+
+## Quick Start
+
+### Deploy Milvus
+
+To accommodate the variety of requirements, Milvus offers as many as four deployment methods. PyMilvus supports Milvus deployed with any of the methods below:
+
+1. [Build from source code](https://github.com/milvus-io/milvus#to-start-developing-milvus)
+2. Install with Docker Compose 
+   - [standalone](https://milvus.io/docs/install_standalone-docker.md)
+   - [cluster](https://milvus.io/docs/v2.0.0/install_cluster-docker.md)
+
+3. Install on Kunernetes 
+   - [standalone](https://milvus.io/docs/install_standalone-helm.md)
+   - [cluster](https://milvus.io/docs/v2.0.0/install_cluster-helm.md)
+
+4. Install with KinD
+
+> For test purposes, we recommend installing Milvus with KinD. KinD supports the ClickOnce deployment of Milvus and its test client. KinD deployment is tailored for scenarios with small data scale, such as development/debugging test cases and functional verification.
+
+- **Prerequisites**
+
+   - [Docker](https://docs.docker.com/get-docker/) (19.05 or higher)
+   - [Docker Compose](https://docs.docker.com/compose/install/) (1.25.5 or higher)
+   - [jq](https://stedolan.github.io/jq/download/) (1.3 or higher)
+   - [kubectl](https://kubernetes.io/docs/tasks/tools/) (1.14 or higher)
+   - [Helm](https://helm.sh/docs/intro/install/) (3.0 or higher)
+   - [KinD](https://kind.sigs.k8s.io/docs/user/quick-start/#installation) (0.10.0 or higher)
+
+- **Install KinD with script**
+
+   1. Enter the local directory of the code ***/milvus/tests/scripts/**
+   2. Build the KinD environment, and execute CI Regression test cases automatically:
+
+   ```shell
+   $ ./e2e-k8s.sh 
+   ```
+
+   - By default, KinD environment will be automatically cleaned up after the execution of the test case. If you need to keep the KinD environment:
+
+   ```shell
+   $ ./e2e-k8s.sh --skip-cleanup
+   ```
+
+   - Skip the automatic test case execution and keep the KinD environment:
+
+   ```shell
+   $ ./e2e-k8s.sh --skip-cleanup --skip-test --manual
+   ```
+
+> Note: You need to log in to the containers of the test client to proceed manual execution and debugging of the test case.
+
+   - See more script parameters:
+
+   ```shell
+$ ./e2e-k8s.sh --help
+   ```
+
+   - Export cluster logs:
+
+   ```shell
+$ kind export logs .
+   ```
+
+
+
+### PyMilvus Test Environment Deployment and Case Execution
+
+We recommend using Python 3.12, consistent with the Python client CI runtime.
+
+> Note: Procedures listed below will be completed automatically if you deployed Milvus using KinD.
+
+1. Install the Python package prerequisite for the test, enter ***/milvus/tests/python_client/**, and execute:
+
+   ```bash
+   $ pip install -r requirements.txt
+   ```
+
+2. The default test log path is **/tmp/ci_logs/** under the **config** directory. You can add environment variables to change the path before booting up test cases:
+
+   ```bash
+   $ export CI_LOG_PATH=/tmp/ci_logs/test/
+   ```
+
+| **Log Level** | **Log File**      |
+| ------------- | ----------------- |
+| `debug`       | ci_test_log.debug |
+| `info`        | ci_test_log.log   |
+| `error`       | ci_test_log.err   |
+
+1. You can configure default parameters in **pytest.ini** under the root path. For instance:
+
+   ```python
+   addopts = --host *.*.*.*  --html=/tmp/ci_logs/report.html
+   ```
+
+where `host` should be set as the IP address of the Milvus service, and `*.html` is the report generated for the test.
+
+2. Enter **testcases** directory, run following command, which is consistent with the command under the pytest framework, to execute the test case:
+
+   ```bash
+   $ python3 -W ignore -m pytest <test_file_name>
+   ```
+
+## An Introduction to Test Modules
+
+### Module Overview
+
+![Module Overview](./graphs/module_call_diagram.jpeg)
+
+<img src="graphs/sdk_test_flow_chart.jpg" alt="SDK Test Flow Chart"/>
+
+### Working directories and files
+
+- **base**: stores the encapsulated **PyMilvus** **module** files, and setup & teardown functions for pytest framework.
+- **check**: stores the **check module** files for returned results from interface.
+- **common**: stores the files of **common methods and parameters** for test cases.
+- **config**: stores the **basic configuration file.**
+- **testcases**: stores **test case scripts.**
+- **utils**: stores **utility programs**, such as utility log and environment detection methods.
+- **requirements.txt**: specifies the **python package** required for executing test cases
+- **conftest.py**: you can compile fixture functions or local plugins in this file. These functions and plugins implement within the current folder and its subfolder.
+- **pytest.ini**: the main configuration file for pytest.
+
+### Critical design ideas
+
+- **base/\*_wrapper.py** encapsulates the tested interface, uniformly processes requests from the interface, abstracts the returned results, and passes the results to **check/func_check.py** module for checking.
+- **check/func_check.py** encompasses result checking methods for each interface for invocation from test cases.
+- **base/client_base.py** uses pytest framework to process setup/teardown functions correspondingly.
+- Test case files in **testcases** folder should be compiled inheriting the **TestcaseBase** module from **base/client_base.py**.  Compile the common methods and parameters used by test cases into the **Common** module for invocation.
+- Add global configurations under **config** directory, such as log path, etc.
+- Add global implementation methods under **utils** directory, such as utility log module.
+
+### Adding codes
+
+This section specifies references while adding new test cases or framework tools.
+
+#### Notice and best practices
+
+1. Coding style
+
+- Test files: each SDK category corresponds to a test file. So do `load` and `search` methods.
+
+- Test categories: test files fall into two categories
+
+  - `TestObjectParams`:
+    - Indicates the parameter test of corresponding interface. For instance, `TestPartitionParams` represents the parameter test for Partition interface.
+    - Tests the target category/method under different parameter inputs. The parameter test will cover `default`, `empty`, `none`, `datatype`,  `maxsize`, etc.
+  - `TestObjectOperations`:
+    - Indicates the function/operation test of corresponding interface. For instance, `TestPartitionOperations` represents the function/operation test for Partition interface.
+    - Tests the target category/method with legit parameter inputs and interaction with other interfaces.
+
+- Testcase naming
+
+  - `TestObjectParams`:
+
+    - Name after the parameter input of the test case. For instance, `test_partition_empty_name()` represents test on performance with the empty string as the `name` parameter input.
+
+  - `TestObjectOperations`
+
+    - Name after the operation procedure of the test case. For instance, `test_partition_drop_partition_twice()` represents the test on the performance when dropping partitions twice consecutively.
+    - Name after assertions. For instance, `test_partition_maximum_partitions()` represents test on the maximum number of partitions that can be created.
+
+2. Notice
+
+- Do not initialize PyMilvus objects in the test case files.
+- Generally, do not add log IDs to test case files.
+- Directly call the encapsulated methods or attributes in test cases, as shown below:
+
+> To create multiple partitions objects, call `self.init_partition_wrap()`, which returns the newly created partition objects. Call `self.partition_wrap` instead when you do not need multiple objects.
+
+   ```python
+# create partition  -Call the default initialization method
+partition_w = self.init_partition_wrap()
+assert partition_w.is_empty
+   ```
+
+   ```python
+# create partition    -Directly call the encapsulated object
+self.partition_wrap.init_partition(collection=collection_name, name=partition_name)
+assert self.partition_wrap.is_empty
+   ```
+
+- To test on the error or exception returned from interfaces:
+  - Call `check_task=CheckTasks.err_res`.
+  - Input the expected error ID and message.
+
+   ```python
+  # create partition with collection is None
+  self.partition_wrap.init_partition(collection=None, name=partition_name, check_task=CheckTasks.err_res, check_items={ct.err_code: 1, ct.err_msg: "'NoneType' object has no attribute"})
+   ```
+
+- To test on the normal value returned from interfaces:
+  - Call `check_task=CheckTasks.check_partition_property`. You can build new test methods in `CheckTasks` for invocation in test cases.
+  - Input the expected result for test methods.
+
+   ```python
+  # create partition
+  partition_w = self.init_partition_wrap(collection_w, partition_name, check_task=CheckTasks.check_partition_property, check_items={"name": partition_name, "description": description, "is_empty": True, "num_entities": 0})
+   ```
+
+3. Adding test cases
+
+- Find the encapsulated tested interface with the same name in the ***_wrapper.py** files under **base** directory. Each interface returns a list with two values, among which one is interface returned results of PyMilvus, and the other is the assertion of normal/abnormal results, i.e. `True`/`False`. The returned judgment can be used in the extra result checking of test cases.
+- Add the test cases in the corresponding test file of the tested interface in **testcases** folder. You can refer to all test files under this directory to create your own test cases as shown below:
+
+   ```python
+   @pytest.mark.tags(CaseLabel.L1)
+   @pytest.mark.parametrize("partition_name", [cf.gen_unique_str(prefix)])
+   def test_partition_dropped_collection(self, partition_name):
+       """
+       target: verify create partition against a dropped collection
+       method: 1. create collection1
+               2. drop collection1
+               3. create partition in collection1
+       expected: raise exception
+       """
+       # create collection
+       collection_w = self.init_collection_wrap()
+       # drop collection
+       collection_w.drop()
+       # create partition failed
+       self.partition_wrap.init_partition(collection_w.collection, partition_name, check_task=CheckTasks.err_res, check_items={ct.err_code: 4, ct.err_msg: "collection not found"})
+   ```
+
+- Tips
+   - Case comments encompass three parts: object, test method, and expected result. You should specify each part.
+   - Initialize the tested category in the setup method of the Base category in the **base/client_base.py** file, as shown below:
+   ```python
+   self.connection_wrap = ApiConnectionsWrapper()
+   self.utility_wrap = ApiUtilityWrapper()
+   self.collection_wrap = ApiCollectionWrapper()
+   self.partition_wrap = ApiPartitionWrapper()
+   self.index_wrap = ApiIndexWrapper()
+   self.collection_schema_wrap = ApiCollectionSchemaWrapper()
+   self.field_schema_wrap = ApiFieldSchemaWrapper()
+   ```
+   - Pass the parameters with corresponding encapsulated methods when calling the interface you need to test on. As shown below, align all parameters with those in PyMilvus interfaces except for `check_task` and `check_items`.
+   ```python
+   def init_partition(self, collection, name, description="", check_task=None, check_items=None, **kwargs)
+   ```
+   - `check_task` is used to select the corresponding interface test method in the ResponseChecker check category in the **check/func_check.py** file. You can choose methods under the `CheckTasks` category in the **common/common_type.py** file.
+   - The specific content of `check_items` passed to the test method is determined by the implemented test method `check_task`.
+   - The tested interface can return normal results when `CheckTasks` and `check_items` are not passed.
+
+4. Adding framework functions
+
+- Add global methods or tools under **utils** directory.
+
+- Add corresponding configurations under **config** directory.

tests/python_client/README_CN.md

@@ -0,0 +1,270 @@
+# 测试框架使用指南
+
+## 简介
+
+基于 pytest 编写的 **PyMilvus** 的测试框架。
+
+**测试代码：** https://github.com/milvus-io/milvus/tree/master/tests/python_client
+
+## 快速开始
+
+### 部署 Milvus
+
+Milvus 支持4种部署方式，请根据需求选择部署方式，PyMilvus 支持任意部署下的 Milvus。
+
+* [源码编译部署](https://github.com/milvus-io/milvus/blob/master/DEVELOPMENT.md)
+* Docker Compose 部署([单机版本](https://milvus.io/docs/zh/install_standalone-docker.md) [分布式版本](https://milvus.io/docs/zh/install-overview.md))
+* Kubernetes 部署([单机版本](https://milvus.io/docs/zh/install-overview.md) [分布式版本](https://milvus.io/docs/zh/install_cluster-helm.md))
+* KinD 部署
+
+    KinD部署提供一键安装部署：最新的Milvus服务和测试客户端。KinD部署非常适合开发/调试测试用例，功能验证等对数据规模要求不大的场景，但并不适合性能或压力等有较大数据规模的场景。
+
+    1. 准备环境
+
+    2. [安装 Docker 、Docker Compose、jq、kubectl、helm、kind](https://github.com/milvus-io/milvus/blob/master/tests/README.md)
+
+    3. 脚本安装
+
+        - 进入代码目录 `*/milvus/tests/scripts/`
+
+        - 新建KinD环境，并自动执行CI Regression测试用例：
+
+            ```shell
+            ./e2e-k8s.sh 
+            ```
+
+            > **_NOTE:_**  默认参数下KinD环境将在执行完测试用例后被自动清理
+            
+
+        - 如果需保留KinD环境，请使用--skip-cleanup参数:
+
+            ```shell
+            ./e2e-k8s.sh --skip-cleanup
+            ```
+
+        - 如不需要自动执行测试用例，并保留KinD环境：
+
+            ```shell
+            ./e2e-k8s.sh --skip-cleanup --skip-test --manual
+            ```
+
+            > **_NOTE:_**  需要login到测试客户端的container进行手动执行或调试测试用例
+
+        - 更多脚本运行参数，请使用--help查看：
+
+            ```shell
+            ./e2e-k8s.sh --help
+            ```
+
+        - 导出集群日志
+
+            ```shell
+            kind export logs .
+            ```
+
+
+### PyMilvus 测试环境部署及用例执行
+
+推荐使用 **Python 3.12**，与 Python client CI 运行环境保持一致。
+
+> **_NOTE:_**  如选择KinD部署方式，以下步骤可以自动完成
+
+1. 安装测试所需的 Python 包，进入代码 `*/milvus/tests/python_client/` 目录，执行命令：
+
+    ```shell
+    pip install -r requirements.txt
+    ```
+
+2. 在`config`目录下，测试的日志目录默认为：`/tmp/ci_logs/`，可在启动测试用例之前添加环境变量来修改log的存放路径：
+
+    ```shell
+    export CI_LOG_PATH=/tmp/ci_logs/test/
+    ```
+
+    | **Log Level** | **Log File**      |
+    | ------------- | ----------------- |
+    | Debug         | ci_test_log.debug |
+    | Info          | ci_test_log.log   |
+    | Error         | ci_test_log.err   |
+
+3. 在主目录 `pytest.ini` 文件内可设置默认传递的参数，如下例中 ip 为所需要设置的 milvus 服务的ip地址，`*.html` 为测试生成的 `report`：
+
+    ```shell
+    addopts = --host *.*.*.*  --html=/tmp/ci_logs/report.html
+    ```
+
+4. 进入 `testcases` 目录，命令与 [pytest](https://docs.pytest.org/en/6.2.x/) 框架的执行命令一致，运行如下命令执行测试用例：
+
+    ```shell
+    python3 -W ignore -m pytest <选择的测试文件>
+    ```
+
+
+
+## 模块介绍
+
+**模块调用关系图**
+
+[![img](https://github.com/milvus-io/milvus/raw/master/tests/python_client/graphs/module_call_diagram.jpeg)](https://github.com/milvus-io/milvus/blob/master/tests/python_client/graphs/module_call_diagram.jpeg)
+
+<img src="graphs/sdk_test_flow_chart.jpg" alt="SDK Test Flow Chart"/>
+
+### 工作目录及文件介绍
+
+- **base**：放置已封装好的 **PyMilvus 模块文件**，以及 Pytest 框架的 setup 和 teardown 处理等
+- **check**：接口返回结果的**检查模块**
+- **common**：测试用例**通用的方法和参数**
+- **config**：**基础配置**内容
+- **testcases**：存放**测试用例**脚本
+- **utils**：**通用程序**，如可用于全局的日志类、或者环境检查的方法等
+- **requirements**：执行测试文件所**依赖**的 python 包
+- **conftest.py**：编写**装饰器函数**，或者自己实现的**本地插件**，作用范围为该文件存放的目录及其子目录
+- **pytest.ini**：pytest 的主**配置**文件
+
+ 
+
+### 主要设计思路
+
+- **base/\*_warpper.py**: **封装被测接口**，统一处理接口请求，提取接口请求的返回结果，传入 **check/func_check.py** 模块进行结果检查。
+- **check/func_check.py**: 模块编写各接口返回结果的检查方法，供测试用例调用。
+- **base/client_base.py**: 模块使用pytest框架，进行相应的setup/teardown方法处理。
+- **testcases** 目录下的测试文件，继承 **base/client_base.py** 里的 **TestcaseBase** 模块，进行相应的测试用例编写。用例里用到的通用参数和数据处理方法，写入**common**模块供用例调用。
+- **config** 目录下加入一些全局配置，如日志的路径。
+- **utils** 目录下负责实现全局的方法，如全局可用的日志模块。
+
+ 
+
+## 代码添加
+
+可参考添加新的测试用例或框架工具。
+
+ 
+
+### Python 测试代码添加注意事项
+
+#### 测试编码风格
+
+- test 文件：每一个 SDK 类对应一个 test 文件，Load 和 Search 单独对应一个 test 文件
+
+- test类：每一个 test 文件中分两个类
+
+    - TestObjectParams ：
+        - 如 TestPartitionParams 表示 Partition Interface 参数检查测试用例类
+        - 检查在不同输入参数条件下，目标类/方法的表现，参数注意覆盖default，empty，none，datatype，maxsize边界值等
+
+    - TestObjectOperations:
+        - 如 TestPartitionOperations 表示 Partition Interface 针对不同 function 或操作的测试
+        - 检查在合法输入参数，与其他接口有一定交互的条件下，目标类/方法的返回和表现
+
+- testcase 命名
+    - TestObjectParams 类
+        - 以testcase输入参数区分命名，如 test_partition_empty_name() 表示验证空字符串作为 name 参数输入的表现
+    - TestObjectOperations 类
+        - 以 testcase 操作步骤区分命名，如 test_partition_drop_partition_twice() 表示验证连续 drop 两次 partition 的表现
+        - 以 testcase 验证点区分命名，如 test_partition_maximum_partitions() 表示验证创建 partition 的最大数量
+
+
+#### 编码注意事项
+
+- 不能在测试用例文件中初始化 PyMilvus 对象
+    - 一般情况下，不在测试用例文件中直接添加日志代码
+    - 在测试用例中，应直接调用封装好的方法或者属性，如下所示：
+ 
+        > **_NOTE:_** 如当需要创建多个 partition 对象时，可调用方法 self.init_partition_wrap()，该方法返回的结果就是新生成的 partition 对象。当无需创建多个对象时，直接使用 self.partition_wrap 即可。
+
+        ```python
+        # create partition  -Call the default initialization method
+        partition_w = self.init_partition_wrap()
+        assert partition_w.is_empty
+        ```
+
+        ```python
+        # create partition    -Directly call the encapsulated object
+        self.partition_wrap.init_partition(collection=collection_name, name=partition_name)
+        assert self.partition_wrap.is_empty
+        ```
+
+- 验证接口返回错误或异常
+    - check_task=CheckTasks.err_res
+    - 输入期望的错误码和错误信息
+
+        ```python
+        # create partition with collection is None
+        self.partition_wrap.init_partition(collection=None, name=partition_name, check_task=CheckTasks.err_res, check_items={ct.err_code: 1, ct.err_msg: "'NoneType' object has no attribute"})
+        ```
+
+- 验证接口返回正常返回值
+    - check_task=CheckTasks.check_partition_property，可以在 CheckTasks 中新建校验方法，在用例中调用使用
+    - 输入期望的结果，供校验方法使用
+
+        ```python
+        # create partition
+        partition_w = self.init_partition_wrap(collection_w, partition_name, check_task=CheckTasks.check_partition_property, check_items={"name": partition_name, "description": description, "is_empty": True, "num_entities": 0})
+        ```
+
+#### 测试用例添加
+
+- 在 base 文件夹的 wrapper 文件底下找到封装好的同名被测接口，各接口返回2个值的list，第一个是 PyMilvus 的接口返回结果，第二个是接口返回结果正常/异常的判断，为True/False。该返回可用于在用例中做额外的结果检查。
+- 在 testcases 文件夹下找到被测接口相应的测试文件，进行用例添加。如下所示，全部测试用例可直接参考 testcases 目录下的所有 test 文件：
+
+    ```python
+    @pytest.mark.tags(CaseLabel.L1)
+    @pytest.mark.parametrize("partition_name", [cf.gen_unique_str(prefix)])
+    def test_partition_dropped_collection(self, partition_name):
+        """
+        target: verify create partition against a dropped collection
+        method: 1. create collection1
+                2. drop collection1
+                3. create partition in collection1
+        expected: 1. raise exception
+        """
+
+        # create collection
+        collection_w = self.init_collection_wrap()
+
+        # drop collection
+        collection_w.drop()
+
+        # create partition failed
+        self.partition_wrap.init_partition(
+                                            collection_w.collection, 
+                                            partition_name, 
+                                            check_task=CheckTasks.err_res, 
+                                            check_items={
+                                                        ct.err_code: 1, 
+                                                        ct.err_msg: "can't find collection"})
+    ```
+
+- Tips
+
+    - 用例注释分为三个部分：目标，测试方法及期望结果，依此说明相应内容
+    - 在 base/client_base.py 文件 Base 类的 setup 方法中对被测试的类进行了初始化，如下图所示：
+
+        ```python
+        self.connection_wrap = ApiConnectionsWrapper()
+        self.utility_wrap = ApiUtilityWrapper()
+        self.collection_wrap = ApiCollectionWrapper()
+        self.partition_wrap = ApiPartitionWrapper()
+        self.index_wrap = ApiIndexWrapper()
+        self.collection_schema_wrap = ApiCollectionSchemaWrapper()
+        self.field_schema_wrap = ApiFieldSchemaWrapper()
+        ```
+
+    - 调用需要测试的接口，应按照相应封装好的方法传入参数。如下所示，除了 check_task，check_items 两个参数外，其余参数与 PyMilvus 的接口参数一致。
+
+        ```python
+        def init_partition(self, collection, name, description="", check_task=None, check_items=None, **kwargs)
+        ```
+
+    - check_task 用来选择 check/func_check.py 文件中 ResponseChecker 检查类中对应的接口检查方法，可选择的方法在 common/common_type.py 文件的 CheckTasks 类中。
+    - check_items 传入检查方法所需的特定内容，具体内容由实现的检查方法所决定。
+    - 默认不传这两个参数，则检查接口能正常返回请求结果。
+
+#### 框架功能添加
+
+- 在 utils 目录下添加需要的全局方法或者工具
+- 可将相应的配置内容加入 config 目录下
+
+
+
+

tests/python_client/assets/ann_hdf5/download.sh

@@ -0,0 +1,18 @@
+#!/bin/bash
+# refer to https://github.com/yahoojapan/gongt/blob/master/assets/bench/download.sh
+
+function check () {
+    if [ ! -e $1 ]; then
+        curl -LO $2
+    fi
+    # md5sum -c $1.md5
+}
+
+# check fashion-mnist-784-euclidean.hdf5 http://vectors.erikbern.com/fashion-mnist-784-euclidean.hdf5
+# check glove-25-angular.hdf5 http://vectors.erikbern.com/glove-25-angular.hdf5
+# check glove-50-angular.hdf5 http://vectors.erikbern.com/glove-50-angular.hdf5
+# check glove-100-angular.hdf5 http://vectors.erikbern.com/glove-100-angular.hdf5
+# check glove-200-angular.hdf5 http://vectors.erikbern.com/glove-200-angular.hdf5
+# check mnist-784-euclidean.hdf5 http://vectors.erikbern.com/mnist-784-euclidean.hdf5
+# check nytimes-256-angular.hdf5 http://vectors.erikbern.com/nytimes-256-angular.hdf5
+check sift-128-euclidean.hdf5 http://vectors.erikbern.com/sift-128-euclidean.hdf5

tests/python_client/base/async_milvus_client_wrapper.py

@@ -0,0 +1,267 @@
+import asyncio
+import sys
+from typing import Optional, List, Union, Dict
+
+from pymilvus import (
+    AsyncMilvusClient,
+    AnnSearchRequest,
+    RRFRanker,
+)
+from pymilvus.orm.types import CONSISTENCY_STRONG
+from pymilvus.orm.collection import CollectionSchema
+
+from check.func_check import ResponseChecker
+from utils.api_request import api_request, logger_interceptor
+
+
+class AsyncMilvusClientWrapper:
+    async_milvus_client = None
+
+    def __init__(self, active_trace=False):
+        self.active_trace = active_trace
+
+    def init_async_client(self, uri: str = "http://localhost:19530",
+                          user: str = "",
+                          password: str = "",
+                          db_name: str = "",
+                          token: str = "",
+                          timeout: Optional[float] = None,
+                          active_trace=False,
+                          check_task=None, check_items=None,
+                          **kwargs):
+        self.active_trace = active_trace
+
+        """ In order to distinguish the same name of collection """
+        func_name = sys._getframe().f_code.co_name
+        res, is_succ = api_request([AsyncMilvusClient, uri, user, password, db_name, token,
+                                    timeout], **kwargs)
+        self.async_milvus_client = res if is_succ else None
+        check_result = ResponseChecker(res, func_name, check_task, check_items, is_succ, **kwargs).run()
+        return res, check_result
+
+    @logger_interceptor()
+    async def list_collections(self, timeout: Optional[float] = None, **kwargs):
+        return await self.async_milvus_client.list_collections(timeout, **kwargs)
+    
+    @logger_interceptor()
+    async def has_collection(self, collection_name: str, timeout: Optional[float] = None, **kwargs):
+        return await self.async_milvus_client.has_collection(collection_name, timeout, **kwargs)
+    
+    @logger_interceptor()
+    async def has_partition(self, collection_name: str, partition_name: str, timeout: Optional[float] = None, **kwargs):
+        return await self.async_milvus_client.has_partition(collection_name, partition_name, timeout, **kwargs)
+
+    @logger_interceptor()
+    async def describe_collection(self, collection_name: str, timeout: Optional[float] = None, **kwargs):
+        return await self.async_milvus_client.describe_collection(collection_name, timeout, **kwargs)
+
+    @logger_interceptor()
+    async def list_partitions(self, collection_name: str, timeout: Optional[float] = None, **kwargs):
+        return await self.async_milvus_client.list_partitions(collection_name, timeout, **kwargs)
+
+    @logger_interceptor()
+    async def get_collection_stats(self, collection_name: str, timeout: Optional[float] = None, **kwargs):
+        return await self.async_milvus_client.get_collection_stats(collection_name, timeout, **kwargs)
+
+    @logger_interceptor()
+    async def flush(self, collection_name: str, timeout: Optional[float] = None, **kwargs):
+        return await self.async_milvus_client.flush(collection_name, timeout, **kwargs)
+
+    @logger_interceptor()
+    async def get_load_state(self, collection_name: str, timeout: Optional[float] = None, **kwargs):
+        return await self.async_milvus_client.get_load_state(collection_name, timeout, **kwargs)
+
+    @logger_interceptor()
+    async def describe_index(self, collection_name: str, index_name: str, timeout: Optional[float] = None, **kwargs):
+        return await self.async_milvus_client.describe_index(collection_name, index_name, timeout, **kwargs)
+
+    @logger_interceptor()
+    async def create_database(self, db_name: str, timeout: Optional[float] = None, **kwargs):
+        return await self.async_milvus_client.create_database(db_name, timeout, **kwargs)
+
+    @logger_interceptor()
+    async def drop_database(self, db_name: str, timeout: Optional[float] = None, **kwargs):
+        return await self.async_milvus_client.drop_database(db_name, timeout, **kwargs)
+
+    @logger_interceptor()
+    async def list_databases(self, timeout: Optional[float] = None, **kwargs):
+        return await self.async_milvus_client.list_databases(timeout, **kwargs)
+    
+    @logger_interceptor()
+    async def list_indexes(self, collection_name: str, field_name: str = "", **kwargs):
+        return await self.async_milvus_client.list_indexes(collection_name, field_name, **kwargs)
+
+    @logger_interceptor()
+    async def create_collection(self,
+                                collection_name: str,
+                                dimension: Optional[int] = None,
+                                primary_field_name: str = "id",  # default is "id"
+                                id_type: str = "int",  # or "string",
+                                vector_field_name: str = "vector",  # default is  "vector"
+                                metric_type: str = "COSINE",
+                                auto_id: bool = False,
+                                timeout: Optional[float] = None,
+                                schema: Optional[CollectionSchema] = None,
+                                index_params=None,
+                                **kwargs):
+        kwargs["consistency_level"] = kwargs.get("consistency_level", CONSISTENCY_STRONG)
+
+        return await self.async_milvus_client.create_collection(collection_name, dimension,
+                                                                primary_field_name,
+                                                                id_type, vector_field_name, metric_type,
+                                                                auto_id,
+                                                                timeout, schema, index_params, **kwargs)
+
+    @logger_interceptor()
+    async def drop_collection(self, collection_name: str, timeout: Optional[float] = None, **kwargs):
+        return await self.async_milvus_client.drop_collection(collection_name, timeout, **kwargs)
+
+    @logger_interceptor()
+    async def load_collection(self, collection_name: str, timeout: Optional[float] = None, **kwargs):
+        return await self.async_milvus_client.load_collection(collection_name, timeout, **kwargs)
+    
+    @logger_interceptor()
+    async def release_collection(self, collection_name, timeout=None, **kwargs):
+        return await self.async_milvus_client.release_collection(collection_name, timeout, **kwargs)
+
+    @logger_interceptor()
+    async def truncate_collection(self, collection_name: str, timeout: Optional[float] = None, **kwargs):
+        return await self.async_milvus_client.truncate_collection(collection_name, timeout, **kwargs)
+    
+    @logger_interceptor()
+    async def list_persistent_segments(self, collection_name: str, timeout: Optional[float] = None, **kwargs):
+        return await self.async_milvus_client.list_persistent_segments(collection_name, timeout, **kwargs)
+
+    @logger_interceptor()
+    async def create_index(self, collection_name: str, index_params, timeout: Optional[float] = None,
+                           **kwargs):
+        return await self.async_milvus_client.create_index(collection_name, index_params, timeout, **kwargs)
+
+    @logger_interceptor()
+    async def drop_index(self, collection_name, index_name, timeout=None, **kwargs):
+        return await self.async_milvus_client.drop_index(collection_name, index_name, timeout, **kwargs)
+
+    # @logger_interceptor()
+    # async def list_indexes(self, collection_name, field_name="", timeout=None, **kwargs):
+    #     return await self.async_milvus_client.list_indexes(collection_name, field_name, timeout, **kwargs)
+
+    @logger_interceptor()
+    async def create_partition(self, collection_name, partition_name, timeout=None, **kwargs):
+        return await self.async_milvus_client.create_partition(collection_name, partition_name, timeout, **kwargs)
+    
+    @logger_interceptor()
+    async def drop_partition(self, collection_name, partition_name, timeout=None, **kwargs):
+        return await self.async_milvus_client.drop_partition(collection_name, partition_name, timeout, **kwargs)
+    
+    @logger_interceptor()
+    async def load_partitions(self, collection_name, partition_names, timeout=None, **kwargs):
+        return await self.async_milvus_client.load_partitions(collection_name, partition_names, timeout, **kwargs)
+
+    @logger_interceptor()
+    async def release_partitions(self, collection_name, partition_names, timeout=None, **kwargs):
+        return await self.async_milvus_client.release_partitions(collection_name, partition_names, timeout, **kwargs)
+    
+    @logger_interceptor()
+    async def insert(self,
+                     collection_name: str,
+                     data: Union[Dict, List[Dict]],
+                     timeout: Optional[float] = None,
+                     partition_name: Optional[str] = "",
+                     **kwargs):
+        return await self.async_milvus_client.insert(collection_name, data, timeout, partition_name, **kwargs)
+
+    @logger_interceptor()
+    async def upsert(self,
+                     collection_name: str,
+                     data: Union[Dict, List[Dict]],
+                     timeout: Optional[float] = None,
+                     partition_name: Optional[str] = "",
+                     **kwargs):
+        return await self.async_milvus_client.upsert(collection_name, data, timeout, partition_name, **kwargs)
+
+    @logger_interceptor()
+    async def search(self,
+                     collection_name: str,
+                     data: Union[List[list], list],
+                     filter: str = "",
+                     limit: int = 10,
+                     output_fields: Optional[List[str]] = None,
+                     search_params: Optional[dict] = None,
+                     timeout: Optional[float] = None,
+                     partition_names: Optional[List[str]] = None,
+                     anns_field: Optional[str] = None,
+                     **kwargs):
+        return await self.async_milvus_client.search(collection_name, data,
+                                                     filter,
+                                                     limit, output_fields, search_params,
+                                                     timeout,
+                                                     partition_names, anns_field, **kwargs)
+
+    @logger_interceptor()
+    async def hybrid_search(self,
+                            collection_name: str,
+                            reqs: List[AnnSearchRequest],
+                            ranker: RRFRanker,
+                            limit: int = 10,
+                            output_fields: Optional[List[str]] = None,
+                            timeout: Optional[float] = None,
+                            partition_names: Optional[List[str]] = None,
+                            **kwargs):
+        return await self.async_milvus_client.hybrid_search(collection_name, reqs,
+                                                            ranker,
+                                                            limit, output_fields,
+                                                            timeout, partition_names, **kwargs)
+
+    @logger_interceptor()
+    async def query(self,
+                    collection_name: str,
+                    filter: str = "",
+                    output_fields: Optional[List[str]] = None,
+                    timeout: Optional[float] = None,
+                    ids: Optional[Union[List, str, int]] = None,
+                    partition_names: Optional[List[str]] = None,
+                    **kwargs):
+        return await self.async_milvus_client.query(collection_name, filter,
+                                                    output_fields, timeout,
+                                                    ids, partition_names,
+                                                    **kwargs)
+
+    @logger_interceptor()
+    async def get(self,
+                  collection_name: str,
+                  ids: Union[list, str, int],
+                  output_fields: Optional[List[str]] = None,
+                  timeout: Optional[float] = None,
+                  partition_names: Optional[List[str]] = None,
+                  **kwargs):
+        return await self.async_milvus_client.get(collection_name, ids,
+                                                  output_fields, timeout,
+                                                  partition_names,
+                                                  **kwargs)
+
+    @logger_interceptor()
+    async def delete(self,
+                     collection_name: str,
+                     ids: Optional[Union[list, str, int]] = None,
+                     timeout: Optional[float] = None,
+                     filter: Optional[str] = None,
+                     partition_name: Optional[str] = None,
+                     **kwargs):
+        return await self.async_milvus_client.delete(collection_name, ids,
+                                                     timeout, filter,
+                                                     partition_name,
+                                                     **kwargs)
+
+    @classmethod
+    def create_schema(cls, **kwargs):
+        kwargs["check_fields"] = False  # do not check fields for now
+        return CollectionSchema([], **kwargs)
+
+    @classmethod
+    def prepare_index_params(cls, field_name: str = "", **kwargs):
+        res, check = api_request([AsyncMilvusClient.prepare_index_params, field_name], **kwargs)
+        return res, check
+
+    @logger_interceptor()
+    async def close(self, **kwargs):
+        return await self.async_milvus_client.close(**kwargs)

tests/python_client/base/client_base.py

@@ -0,0 +1,649 @@
+import sys
+
+from base.database_wrapper import ApiDatabaseWrapper
+from pymilvus import DefaultConfig
+
+sys.path.append("..")
+import pymilvus
+from base.async_milvus_client_wrapper import AsyncMilvusClientWrapper
+from base.collection_wrapper import ApiCollectionWrapper
+from base.connections_wrapper import ApiConnectionsWrapper
+from base.index_wrapper import ApiIndexWrapper
+from base.partition_wrapper import ApiPartitionWrapper
+from base.schema_wrapper import ApiCollectionSchemaWrapper, ApiFieldSchemaWrapper
+from base.utility_wrapper import ApiUtilityWrapper
+from common import common_func as cf
+from common import common_type as ct
+from common.common_params import IndexPrams
+from pymilvus import DataType, MilvusClient, ResourceGroupInfo, utility
+from utils.util_log import test_log as log
+
+
+class Base:
+    """Initialize class object"""
+
+    connection_wrap = None
+    collection_wrap = None
+    partition_wrap = None
+    index_wrap = None
+    utility_wrap = None
+    collection_schema_wrap = None
+    field_schema_wrap = None
+    database_wrap = None
+    tear_down_collection_names = []
+    tear_down_role_names = []
+    tear_down_user_names = []
+    resource_group_list = []
+    async_milvus_client_wrap = None
+    skip_connection = False
+    skip_global_role_cleanup = False
+
+    def setup_class(self):
+        log.info("[setup_class] Start setup class...")
+
+    def teardown_class(self):
+        log.info("[teardown_class] Start teardown class...")
+
+    def setup_method(self, method):
+        log.info(("*" * 35) + " setup " + ("*" * 35))
+        log.info(f"pymilvus version: {pymilvus.__version__}")
+        log.info(f"[setup_method] Start setup test case {method.__name__}.")
+        self._setup_objects()
+
+    def _setup_objects(self):
+        self.connection_wrap = ApiConnectionsWrapper()
+        self.utility_wrap = ApiUtilityWrapper()
+        self.collection_wrap = ApiCollectionWrapper()
+        self.partition_wrap = ApiPartitionWrapper()
+        self.index_wrap = ApiIndexWrapper()
+        self.collection_schema_wrap = ApiCollectionSchemaWrapper()
+        self.field_schema_wrap = ApiFieldSchemaWrapper()
+        self.database_wrap = ApiDatabaseWrapper()
+        self.async_milvus_client_wrap = AsyncMilvusClientWrapper()
+
+    def teardown_method(self, method):
+        log.info(("*" * 35) + " teardown " + ("*" * 35))
+        log.info(f"[teardown_method] Start teardown test case {method.__name__}...")
+        self._teardown_objects()
+
+    def _teardown_objects(self):
+        # Prioritize uri and token for connection
+        if cf.param_info.param_uri:
+            uri = cf.param_info.param_uri
+        else:
+            uri = "http://" + cf.param_info.param_host + ":" + str(cf.param_info.param_port)
+
+        if cf.param_info.param_token:
+            token = cf.param_info.param_token
+        else:
+            token = (
+                f"{cf.param_info.param_user}:{cf.param_info.param_password}"
+                if cf.param_info.param_user and cf.param_info.param_password
+                else None
+            )
+
+        try:
+            """ Drop collection before disconnect """
+            if not self.connection_wrap.has_connection(alias=DefaultConfig.DEFAULT_USING)[0]:
+                if token:
+                    self.connection_wrap.connect(alias=DefaultConfig.DEFAULT_USING, uri=uri, token=token)
+                else:
+                    self.connection_wrap.connect(alias=DefaultConfig.DEFAULT_USING, uri=uri)
+
+            if self.collection_wrap.collection is not None:
+                if self.collection_wrap.collection.name.startswith("alias"):
+                    log.info(f"collection {self.collection_wrap.collection.name} is alias, skip drop operation")
+                else:
+                    self.collection_wrap.drop(check_task=ct.CheckTasks.check_nothing)
+
+            collection_list = self.utility_wrap.list_collections()[0]
+            for collection_name in self.tear_down_collection_names:
+                if collection_name is not None and collection_name in collection_list:
+                    alias_list = self.utility_wrap.list_aliases(collection_name)[0]
+                    if alias_list:
+                        for alias in alias_list:
+                            self.utility_wrap.drop_alias(alias)
+                    self.utility_wrap.drop_collection(collection_name)
+
+            """ Clean up the rgs before disconnect """
+            rgs_list = self.utility_wrap.list_resource_groups()[0]
+            for rg_name in self.resource_group_list:
+                if rg_name is not None and rg_name in rgs_list:
+                    rg = self.utility_wrap.describe_resource_group(
+                        name=rg_name, check_task=ct.CheckTasks.check_nothing
+                    )[0]
+                    if isinstance(rg, ResourceGroupInfo):
+                        if rg.num_available_node > 0:
+                            self.utility_wrap.transfer_node(
+                                source=rg_name, target=ct.default_resource_group_name, num_node=rg.num_available_node
+                            )
+                        self.utility_wrap.drop_resource_group(rg_name, check_task=ct.CheckTasks.check_nothing)
+
+        except Exception as e:
+            log.debug(str(e))
+
+        if not self.skip_global_role_cleanup:
+            try:
+                """ Drop roles before disconnect """
+                if not self.connection_wrap.has_connection(alias=DefaultConfig.DEFAULT_USING)[0]:
+                    if token:
+                        self.connection_wrap.connect(alias=DefaultConfig.DEFAULT_USING, uri=uri, token=token)
+                    else:
+                        self.connection_wrap.connect(alias=DefaultConfig.DEFAULT_USING, uri=uri)
+
+                role_list = self.utility_wrap.list_roles(False)[0]
+                for role in role_list.groups:
+                    role_name = role.role_name
+                    if role_name not in ["admin", "public"]:
+                        each_role = self.utility_wrap.init_role(name=role_name)[0]
+                        each_role.drop()
+
+            except Exception as e:
+                log.debug(str(e))
+
+        try:
+            """ Delete connection and reset configuration"""
+            res = self.connection_wrap.list_connections()
+            for i in res[0]:
+                self.connection_wrap.remove_connection(i[0])
+
+            # because the connection is in singleton mode, it needs to be restored to the original state after teardown
+            self.connection_wrap.add_connection(
+                default={"host": DefaultConfig.DEFAULT_HOST, "port": DefaultConfig.DEFAULT_PORT}
+            )
+        except Exception as e:
+            log.debug(str(e))
+
+
+class TestcaseBase(Base):
+    """
+    Additional methods;
+    Public methods that can be used for test cases.
+    """
+
+    client = None
+
+    def _connect(self, enable_milvus_client_api=False):
+        """Add a connection and create the connect"""
+        if self.skip_connection:
+            return None
+
+        # Prioritize uri and token for connection
+        if cf.param_info.param_uri:
+            uri = cf.param_info.param_uri
+        else:
+            uri = "http://" + cf.param_info.param_host + ":" + str(cf.param_info.param_port)
+
+        if cf.param_info.param_token:
+            token = cf.param_info.param_token
+        else:
+            token = (
+                f"{cf.param_info.param_user}:{cf.param_info.param_password}"
+                if cf.param_info.param_user and cf.param_info.param_password
+                else None
+            )
+
+        if enable_milvus_client_api:
+            self.connection_wrap.connect(alias=DefaultConfig.DEFAULT_USING, uri=uri, token=token)
+            res, is_succ = self.connection_wrap.MilvusClient(uri=uri, token=token)
+            self.client = MilvusClient(uri=uri, token=token)
+        else:
+            if token:
+                res, is_succ = self.connection_wrap.connect(
+                    alias=DefaultConfig.DEFAULT_USING, uri=uri, token=token, secure=cf.param_info.param_secure
+                )
+            else:
+                res, is_succ = self.connection_wrap.connect(alias=DefaultConfig.DEFAULT_USING, uri=uri)
+
+            self.client = MilvusClient(uri=uri, token=token)
+        server_version = utility.get_server_version()
+        log.info(f"server version: {server_version}")
+        return res
+
+    def get_tokens_by_analyzer(self, text, analyzer_params):
+        if cf.param_info.param_uri:
+            uri = cf.param_info.param_uri
+        else:
+            uri = "http://" + cf.param_info.param_host + ":" + str(cf.param_info.param_port)
+
+        client = MilvusClient(uri=uri, token=cf.param_info.param_token)
+        res = client.run_analyzer(text, analyzer_params, with_detail=True, with_hash=True)
+        tokens = [r["token"] for r in res.tokens]
+        return tokens
+
+    # def init_async_milvus_client(self):
+    #     uri = cf.param_info.param_uri or f"http://{cf.param_info.param_host}:{cf.param_info.param_port}"
+    #     kwargs = {
+    #         "uri": uri,
+    #         "user": cf.param_info.param_user,
+    #         "password": cf.param_info.param_password,
+    #         "token": cf.param_info.param_token,
+    #     }
+    #     self.async_milvus_client_wrap.init_async_client(**kwargs)
+
+    def init_collection_wrap(
+        self,
+        name=None,
+        schema=None,
+        check_task=None,
+        check_items=None,
+        enable_dynamic_field=False,
+        with_json=True,
+        **kwargs,
+    ):
+        name = cf.gen_collection_name_by_testcase_name(2) if name is None else name
+        schema = (
+            cf.gen_default_collection_schema(enable_dynamic_field=enable_dynamic_field, with_json=with_json)
+            if schema is None
+            else schema
+        )
+        if not self.connection_wrap.has_connection(alias=DefaultConfig.DEFAULT_USING)[0]:
+            self._connect()
+        collection_w = ApiCollectionWrapper()
+        collection_w.init_collection(name=name, schema=schema, check_task=check_task, check_items=check_items, **kwargs)
+        self.tear_down_collection_names.append(name)
+        return collection_w
+
+    def init_multi_fields_collection_wrap(self, name=cf.gen_unique_str()):
+        vec_fields = [cf.gen_float_vec_field(ct.another_float_vec_field_name)]
+        schema = cf.gen_schema_multi_vector_fields(vec_fields)
+        collection_w = self.init_collection_wrap(name=name, schema=schema)
+        df = cf.gen_dataframe_multi_vec_fields(vec_fields=vec_fields)
+        collection_w.insert(df)
+        assert collection_w.num_entities == ct.default_nb
+        return collection_w, df
+
+    def init_partition_wrap(
+        self, collection_wrap=None, name=None, description=None, check_task=None, check_items=None, **kwargs
+    ):
+        name = cf.gen_unique_str("partition_") if name is None else name
+        description = cf.gen_unique_str("partition_des_") if description is None else description
+        collection_wrap = self.init_collection_wrap() if collection_wrap is None else collection_wrap
+        partition_wrap = ApiPartitionWrapper()
+        partition_wrap.init_partition(
+            collection_wrap.collection, name, description, check_task=check_task, check_items=check_items, **kwargs
+        )
+        return partition_wrap
+
+    def insert_data_general(
+        self,
+        prefix="test",
+        insert_data=False,
+        nb=ct.default_nb,
+        partition_num=0,
+        is_binary=False,
+        is_all_data_type=False,
+        auto_id=False,
+        dim=ct.default_dim,
+        primary_field=ct.default_int64_field_name,
+        is_flush=True,
+        name=None,
+        enable_dynamic_field=False,
+        with_json=True,
+        **kwargs,
+    ):
+        """ """
+        self._connect()
+        collection_name = cf.gen_unique_str(prefix)
+        if name is not None:
+            collection_name = name
+        vectors = []
+        binary_raw_vectors = []
+        insert_ids = []
+        time_stamp = 0
+        # 1 create collection
+        default_schema = cf.gen_default_collection_schema(
+            auto_id=auto_id,
+            dim=dim,
+            primary_field=primary_field,
+            enable_dynamic_field=enable_dynamic_field,
+            with_json=with_json,
+        )
+        if is_binary:
+            default_schema = cf.gen_default_binary_collection_schema(
+                auto_id=auto_id, dim=dim, primary_field=primary_field
+            )
+        if is_all_data_type:
+            default_schema = cf.gen_collection_schema_all_datatype(
+                auto_id=auto_id,
+                dim=dim,
+                primary_field=primary_field,
+                enable_dynamic_field=enable_dynamic_field,
+                with_json=with_json,
+            )
+        log.info("insert_data_general: collection creation")
+        collection_w = self.init_collection_wrap(name=collection_name, schema=default_schema, **kwargs)
+        pre_entities = collection_w.num_entities
+        if insert_data:
+            collection_w, vectors, binary_raw_vectors, insert_ids, time_stamp = cf.insert_data(
+                collection_w,
+                nb,
+                is_binary,
+                is_all_data_type,
+                auto_id=auto_id,
+                dim=dim,
+                enable_dynamic_field=enable_dynamic_field,
+                with_json=with_json,
+            )
+            if is_flush:
+                collection_w.flush()
+                assert collection_w.num_entities == nb + pre_entities
+
+        return collection_w, vectors, binary_raw_vectors, insert_ids, time_stamp
+
+    def init_collection_general(
+        self,
+        prefix="test",
+        insert_data=False,
+        nb=ct.default_nb,
+        partition_num=0,
+        is_binary=False,
+        is_all_data_type=False,
+        auto_id=False,
+        dim=ct.default_dim,
+        is_index=True,
+        primary_field=ct.default_int64_field_name,
+        is_flush=True,
+        name=None,
+        enable_dynamic_field=False,
+        with_json=True,
+        random_primary_key=False,
+        multiple_dim_array=[],
+        is_partition_key=None,
+        vector_data_type=DataType.FLOAT_VECTOR,
+        nullable_fields={},
+        default_value_fields={},
+        language=None,
+        **kwargs,
+    ):
+        """
+        target: create specified collections
+        method: 1. create collections (binary/non-binary, default/all data type, auto_id or not)
+                2. create partitions if specified
+                3. insert specified (binary/non-binary, default/all data type) data
+                   into each partition if any
+                4. not load if specifying is_index as True
+                5. enable insert null data: nullable_fields = {"nullable_fields_name": null data percent}
+                6. enable insert default value: default_value_fields = {"default_fields_name": default value}
+        expected: return collection and raw data, insert ids
+        """
+        log.info("Test case of search interface: initialize before test case")
+        if not self.connection_wrap.has_connection(alias=DefaultConfig.DEFAULT_USING)[0]:
+            self._connect()
+        collection_name = cf.gen_collection_name_by_testcase_name(2)
+        if name is not None:
+            collection_name = name
+        if not isinstance(nullable_fields, dict):
+            log.error("nullable_fields should a dict like {'nullable_fields_name': null data percent}")
+            assert False
+        if not isinstance(default_value_fields, dict):
+            log.error("default_value_fields should a dict like {'default_fields_name': default value}")
+            assert False
+        vectors = []
+        binary_raw_vectors = []
+        insert_ids = []
+        time_stamp = 0
+        # 1 create collection
+        default_schema = cf.gen_default_collection_schema(
+            auto_id=auto_id,
+            dim=dim,
+            primary_field=primary_field,
+            enable_dynamic_field=enable_dynamic_field,
+            with_json=with_json,
+            multiple_dim_array=multiple_dim_array,
+            is_partition_key=is_partition_key,
+            vector_data_type=vector_data_type,
+            nullable_fields=nullable_fields,
+            default_value_fields=default_value_fields,
+        )
+        if is_binary:
+            default_schema = cf.gen_default_binary_collection_schema(
+                auto_id=auto_id,
+                dim=dim,
+                primary_field=primary_field,
+                nullable_fields=nullable_fields,
+                default_value_fields=default_value_fields,
+            )
+        if vector_data_type == DataType.SPARSE_FLOAT_VECTOR:
+            default_schema = cf.gen_default_sparse_schema(
+                auto_id=auto_id,
+                primary_field=primary_field,
+                enable_dynamic_field=enable_dynamic_field,
+                with_json=with_json,
+                multiple_dim_array=multiple_dim_array,
+                nullable_fields=nullable_fields,
+                default_value_fields=default_value_fields,
+            )
+        if is_all_data_type:
+            default_schema = cf.gen_collection_schema_all_datatype(
+                auto_id=auto_id,
+                dim=dim,
+                primary_field=primary_field,
+                enable_dynamic_field=enable_dynamic_field,
+                with_json=with_json,
+                multiple_dim_array=multiple_dim_array,
+                nullable_fields=nullable_fields,
+                default_value_fields=default_value_fields,
+            )
+        log.info("init_collection_general: collection creation")
+        collection_w = self.init_collection_wrap(name=collection_name, schema=default_schema, **kwargs)
+        vector_name_list = cf.extract_vector_field_name_list(collection_w)
+        # 2 add extra partitions if specified (default is 1 partition named "_default")
+        if partition_num > 0:
+            cf.gen_partitions(collection_w, partition_num)
+        # 3 insert data if specified
+        if insert_data:
+            collection_w, vectors, binary_raw_vectors, insert_ids, time_stamp = cf.insert_data(
+                collection_w,
+                nb,
+                is_binary,
+                is_all_data_type,
+                auto_id=auto_id,
+                dim=dim,
+                enable_dynamic_field=enable_dynamic_field,
+                with_json=with_json,
+                random_primary_key=random_primary_key,
+                multiple_dim_array=multiple_dim_array,
+                primary_field=primary_field,
+                vector_data_type=vector_data_type,
+                nullable_fields=nullable_fields,
+                language=language,
+            )
+            if is_flush:
+                assert collection_w.is_empty is False
+                assert collection_w.num_entities == nb
+        # 4 create default index if specified
+        if is_index:
+            # This condition will be removed after auto index feature
+            if is_binary:
+                collection_w.create_index(ct.default_binary_vec_field_name, ct.default_bin_flat_index)
+            elif vector_data_type == DataType.SPARSE_FLOAT_VECTOR:
+                for vector_name in vector_name_list:
+                    collection_w.create_index(vector_name, ct.default_sparse_inverted_index)
+            else:
+                if len(multiple_dim_array) == 0 or is_all_data_type is False:
+                    vector_name_list.append(ct.default_float_vec_field_name)
+                for vector_name in vector_name_list:
+                    # Unlike dense vectors, sparse vectors cannot create flat index.
+                    if DataType.SPARSE_FLOAT_VECTOR.name in vector_name:
+                        collection_w.create_index(vector_name, ct.default_sparse_inverted_index)
+                    elif vector_data_type == DataType.INT8_VECTOR:
+                        collection_w.create_index(vector_name, ct.int8_vector_index)
+                    else:
+                        collection_w.create_index(vector_name, ct.default_flat_index)
+
+            collection_w.load()
+
+        return collection_w, vectors, binary_raw_vectors, insert_ids, time_stamp
+
+    def insert_entities_into_two_partitions_in_half(self, half, prefix="query"):
+        """
+        insert default entities into two partitions(partition_w and _default) in half(int64 and float fields values)
+        :param half: half of nb
+        :return: collection wrap and partition wrap
+        """
+        self._connect()
+        collection_name = cf.gen_collection_name_by_testcase_name(2)
+        collection_w = self.init_collection_wrap(name=collection_name)
+        partition_w = self.init_partition_wrap(collection_wrap=collection_w)
+        # insert [0, half) into partition_w
+        df_partition = cf.gen_default_dataframe_data(nb=half, start=0)
+        partition_w.insert(df_partition)
+        # insert [half, nb) into _default
+        df_default = cf.gen_default_dataframe_data(nb=half, start=half)
+        collection_w.insert(df_default)
+        # flush
+        collection_w.num_entities
+        collection_w.create_index(ct.default_float_vec_field_name, index_params=ct.default_flat_index)
+        collection_w.load(partition_names=[partition_w.name, "_default"])
+        return collection_w, partition_w, df_partition, df_default
+
+    def collection_insert_multi_segments_one_shard(
+        self, collection_prefix, num_of_segment=2, nb_of_segment=1, is_dup=True
+    ):
+        """
+        init collection with one shard, insert data into two segments on one shard (they can be merged)
+        :param collection_prefix: collection name prefix
+        :param num_of_segment: number of segments
+        :param nb_of_segment: number of entities per segment
+        :param is_dup: whether the primary keys of each segment is duplicated
+        :return: collection wrap and partition wrap
+        """
+        collection_name = cf.gen_collection_name_by_testcase_name(2)
+        collection_w = self.init_collection_wrap(name=collection_name, shards_num=1)
+
+        for i in range(num_of_segment):
+            start = 0 if is_dup else i * nb_of_segment
+            df = cf.gen_default_dataframe_data(nb_of_segment, start=start)
+            collection_w.insert(df)
+            assert collection_w.num_entities == nb_of_segment * (i + 1)
+        return collection_w
+
+    def init_resource_group(self, name, using="default", timeout=None, check_task=None, check_items=None, **kwargs):
+        if not self.connection_wrap.has_connection(alias=DefaultConfig.DEFAULT_USING)[0]:
+            self._connect()
+        utility_w = ApiUtilityWrapper()
+        res, check_result = utility_w.create_resource_group(
+            name=name, using=using, timeout=timeout, check_task=check_task, check_items=check_items, **kwargs
+        )
+        if res is None and check_result:
+            self.resource_group_list.append(name)
+        return res, check_result
+
+    def init_user_with_privilege(self, privilege_object, object_name, privilege, db_name="default"):
+        """
+        init a user and role, grant privilege to the role with the db, then bind the role to the user
+        :param privilege_object: privilege object: Global, Collection, User
+        :type privilege_object: str
+        :param object_name: privilege object name
+        :type object_name: str
+        :param privilege: privilege
+        :type privilege: str
+        :param db_name: database name
+        :type db_name: str
+        :return: user name, user pwd, role name
+        :rtype: str, str, str
+        """
+        tmp_user = cf.gen_unique_str("user")
+        tmp_pwd = cf.gen_unique_str("pwd")
+        tmp_role = cf.gen_unique_str("role")
+
+        # create user
+        self.utility_wrap.create_user(tmp_user, tmp_pwd)
+
+        # create role
+        self.utility_wrap.init_role(tmp_role)
+        self.utility_wrap.create_role()
+
+        # grant privilege to the role
+        self.utility_wrap.role_grant(
+            object=privilege_object, object_name=object_name, privilege=privilege, db_name=db_name
+        )
+
+        # bind the role to the user
+        self.utility_wrap.role_add_user(tmp_user)
+
+        return tmp_user, tmp_pwd, tmp_role
+
+    def build_multi_index(self, index_params: dict[str, IndexPrams], collection_obj: ApiCollectionWrapper = None):
+        collection_obj = collection_obj or self.collection_wrap
+        for k, v in index_params.items():
+            collection_obj.create_index(field_name=k, index_params=v.to_dict, index_name=k)
+        log.info(f"[TestcaseBase] Build all indexes done: {list(index_params.keys())}")
+        return collection_obj
+
+    def drop_multi_index(
+        self, index_names: list[str], collection_obj: ApiCollectionWrapper = None, check_task=None, check_items=None
+    ):
+        collection_obj = collection_obj or self.collection_wrap
+        for n in index_names:
+            collection_obj.drop_index(index_name=n, check_task=check_task, check_items=check_items)
+        log.info(f"[TestcaseBase] Drop all indexes done: {index_names}")
+        return collection_obj
+
+    def show_indexes(self, collection_obj: ApiCollectionWrapper = None):
+        collection_obj = collection_obj or self.collection_wrap
+        indexes = {n.field_name: n.params for n in self.collection_wrap.indexes}
+        log.info(f"[TestcaseBase] Collection: `{collection_obj.name}` index: {indexes}")
+        return indexes
+
+    """ Property """
+
+    @property
+    def all_scalar_fields(self):
+        dtypes = [
+            DataType.INT8,
+            DataType.INT16,
+            DataType.INT32,
+            DataType.INT64,
+            DataType.VARCHAR,
+            DataType.BOOL,
+            DataType.FLOAT,
+            DataType.DOUBLE,
+        ]
+        dtype_names = [f"{n.name}" for n in dtypes] + [f"ARRAY_{n.name}" for n in dtypes] + [DataType.JSON.name]
+        return dtype_names
+
+    @property
+    def all_index_scalar_fields(self):
+        return list(set(self.all_scalar_fields) - {DataType.JSON.name})
+
+    @property
+    def inverted_support_dtype_names(self):
+        return self.all_index_scalar_fields
+
+    @property
+    def inverted_not_support_dtype_names(self):
+        return [DataType.JSON.name]
+
+    @property
+    def bitmap_support_dtype_names(self):
+        dtypes = [DataType.INT8, DataType.INT16, DataType.INT32, DataType.INT64, DataType.BOOL, DataType.VARCHAR]
+        dtype_names = [f"{n.name}" for n in dtypes] + [f"ARRAY_{n.name}" for n in dtypes]
+        return dtype_names
+
+    @property
+    def bitmap_not_support_dtype_names(self):
+        return list(set(self.all_scalar_fields) - set(self.bitmap_support_dtype_names))
+
+
+class TestCaseClassBase(TestcaseBase):
+    """
+    Setup objects on class
+    """
+
+    def setup_class(self):
+        log.info("[setup_class] " + " Start setup class ".center(100, "~"))
+        self._setup_objects(self)
+
+    def teardown_class(self):
+        log.info("[teardown_class]" + " Start teardown class ".center(100, "~"))
+        self._teardown_objects(self)
+
+    def setup_method(self, method):
+        log.info(" setup ".center(80, "*"))
+        log.info(f"[setup_method] Start setup test case {method.__name__}.")
+
+    def teardown_method(self, method):
+        log.info(" teardown ".center(80, "*"))
+        log.info(f"[teardown_method] Start teardown test case {method.__name__}...")

tests/python_client/base/collection_wrapper.py

@@ -0,0 +1,387 @@
+import sys
+import time
+from numpy import NaN
+
+from pymilvus import Collection
+
+sys.path.append("..")
+from check.func_check import ResponseChecker
+from utils.api_request import api_request
+from utils.wrapper import trace
+from utils.util_log import test_log as log
+from pymilvus.orm.types import CONSISTENCY_STRONG
+from common.common_func import param_info
+
+TIMEOUT = 180
+INDEX_NAME = ""
+
+
+# keep small timeout for stability tests
+# TIMEOUT = 5
+
+
+class ApiCollectionWrapper:
+    collection = None
+
+    def __init__(self, active_trace=False):
+        self.active_trace = active_trace
+
+    def init_collection(self, name, schema=None, using="default", check_task=None, check_items=None,
+                        active_trace=False, **kwargs):
+        self.active_trace = active_trace
+        consistency_level = kwargs.get("consistency_level", CONSISTENCY_STRONG)
+        kwargs.update({"consistency_level": consistency_level})
+
+        """ In order to distinguish the same name of collection """
+        func_name = sys._getframe().f_code.co_name
+        res, is_succ = api_request([Collection, name, schema, using], **kwargs)
+        self.collection = res if is_succ else None
+        check_result = ResponseChecker(res, func_name, check_task, check_items, is_succ,
+                                       name=name, schema=schema, using=using, **kwargs).run()
+        return res, check_result
+
+    @property
+    def schema(self):
+        return self.collection.schema
+
+    @property
+    def description(self):
+        return self.collection.description
+
+    @property
+    def name(self):
+        return self.collection.name
+
+    @property
+    def is_empty(self):
+        self.flush()
+        return self.collection.is_empty
+
+    @property
+    def is_empty_without_flush(self):
+        return self.collection.is_empty
+
+    @property
+    def num_entities(self):
+        self.flush()
+        return self.collection.num_entities
+
+    @property
+    def num_shards(self):
+        return self.collection.num_shards
+
+    @property
+    def num_entities_without_flush(self):
+        return self.collection.num_entities
+
+    @property
+    def primary_field(self):
+        return self.collection.primary_field
+
+    @property
+    def aliases(self):
+        return self.collection.aliases
+
+    @trace()
+    def construct_from_dataframe(self, name, dataframe, check_task=None, check_items=None, **kwargs):
+        func_name = sys._getframe().f_code.co_name
+        res, is_succ = api_request([Collection.construct_from_dataframe, name, dataframe], **kwargs)
+        self.collection = res[0] if is_succ else None
+        check_result = ResponseChecker(res, func_name, check_task, check_items, is_succ,
+                                       name=name, dataframe=dataframe, **kwargs).run()
+        return res, check_result
+
+    @trace()
+    def drop(self, check_task=None, check_items=None, **kwargs):
+        timeout = kwargs.get("timeout", TIMEOUT)
+        kwargs.update({"timeout": timeout})
+
+        func_name = sys._getframe().f_code.co_name
+        res, check = api_request([self.collection.drop], **kwargs)
+        check_result = ResponseChecker(res, func_name, check_task, check_items, check, **kwargs).run()
+        return res, check_result
+
+    @trace()
+    def load(self, partition_names=None, replica_number=NaN, timeout=None, check_task=None, check_items=None, **kwargs):
+        timeout = TIMEOUT if timeout is None else timeout
+        replica_number = param_info.param_replica_num if replica_number is NaN else replica_number
+
+        func_name = sys._getframe().f_code.co_name
+        res, check = api_request([self.collection.load, partition_names, replica_number, timeout], **kwargs)
+        check_result = ResponseChecker(res, func_name, check_task, check_items, check,
+                                       partition_names=partition_names, **kwargs).run()
+        return res, check_result
+
+    @trace()
+    def release(self, check_task=None, check_items=None, **kwargs):
+        timeout = kwargs.get("timeout", TIMEOUT)
+        kwargs.update({"timeout": timeout})
+
+        func_name = sys._getframe().f_code.co_name
+        res, check = api_request([self.collection.release], **kwargs)
+        check_result = ResponseChecker(res, func_name, check_task,
+                                       check_items, check, **kwargs).run()
+        return res, check_result
+
+    @trace()
+    def insert(self, data, partition_name=None, check_task=None, check_items=None, **kwargs):
+        timeout = kwargs.get("timeout", TIMEOUT)
+        kwargs.update({"timeout": timeout})
+
+        func_name = sys._getframe().f_code.co_name
+        res, check = api_request([self.collection.insert, data, partition_name], **kwargs)
+        check_result = ResponseChecker(res, func_name, check_task, check_items, check,
+                                       data=data, partition_name=partition_name,
+                                       **kwargs).run()
+        return res, check_result
+
+    @trace()
+    def flush(self, check_task=None, check_items=None, **kwargs):
+        timeout = kwargs.get("timeout", TIMEOUT)
+        kwargs.update({"timeout": timeout})
+
+        func_name = sys._getframe().f_code.co_name
+        res, check = api_request([self.collection.flush], **kwargs)
+        check_result = ResponseChecker(res, func_name, check_task,
+                                       check_items, check, **kwargs).run()
+        return res, check_result
+
+    @trace()
+    def search(self, data=None, anns_field=None, param=None, limit=None, expr=None,
+               partition_names=None, output_fields=None, timeout=None, round_decimal=-1,
+               check_task=None, check_items=None, **kwargs):
+        timeout = TIMEOUT if timeout is None else timeout
+
+        func_name = sys._getframe().f_code.co_name
+        res, check = api_request([self.collection.search, data, anns_field, param, limit,
+                                  expr, partition_names, output_fields, timeout, round_decimal], **kwargs)
+        check_result = ResponseChecker(res, func_name, check_task, check_items, check,
+                                       data=data, anns_field=anns_field, param=param, limit=limit,
+                                       expr=expr, partition_names=partition_names,
+                                       output_fields=output_fields,
+                                       timeout=timeout, **kwargs).run()
+        return res, check_result
+
+    @trace()
+    def hybrid_search(self, reqs, rerank, limit, partition_names=None,
+                      output_fields=None, timeout=None, round_decimal=-1,
+                      check_task=None, check_items=None, **kwargs):
+        timeout = TIMEOUT if timeout is None else timeout
+
+        func_name = sys._getframe().f_code.co_name
+        res, check = api_request([self.collection.hybrid_search, reqs, rerank, limit, partition_names,
+                                  output_fields, timeout, round_decimal], **kwargs)
+        check_result = ResponseChecker(res, func_name, check_task, check_items, check,
+                                       reqs=reqs, rerank=rerank, limit=limit,
+                                       partition_names=partition_names,
+                                       output_fields=output_fields,
+                                       timeout=timeout, **kwargs).run()
+        return res, check_result
+
+    @trace()
+    def search_iterator(self, data=None, anns_field=None, param=None, batch_size=None, limit=-1, expr=None,
+                        partition_names=None, output_fields=None, timeout=None, round_decimal=-1,
+                        check_task=None, check_items=None, **kwargs):
+        timeout = TIMEOUT if timeout is None else timeout
+
+        func_name = sys._getframe().f_code.co_name
+        res, check = api_request([self.collection.search_iterator, data, anns_field, param, batch_size, limit,
+                                  expr, partition_names, output_fields, timeout, round_decimal], **kwargs)
+        check_result = ResponseChecker(res, func_name, check_task, check_items, check,
+                                       data=data, anns_field=anns_field, param=param, limit=limit,
+                                       expr=expr, partition_names=partition_names,
+                                       output_fields=output_fields,
+                                       timeout=timeout, **kwargs).run()
+        return res, check_result
+
+    @trace()
+    def query(self, expr, output_fields=None, partition_names=None, timeout=None, check_task=None, check_items=None,
+              **kwargs):
+        timeout = TIMEOUT if timeout is None else timeout
+
+        func_name = sys._getframe().f_code.co_name
+        res, check = api_request([self.collection.query, expr, output_fields, partition_names, timeout], **kwargs)
+        check_result = ResponseChecker(res, func_name, check_task, check_items, check,
+                                       expression=expr, partition_names=partition_names,
+                                       output_fields=output_fields,
+                                       timeout=timeout, **kwargs).run()
+        return res, check_result
+
+    @trace()
+    def query_iterator(self, batch_size=1000, limit=-1, expr=None, output_fields=None, partition_names=None, timeout=None,
+                       check_task=None, check_items=None, **kwargs):
+        timeout = TIMEOUT if timeout is None else timeout
+
+        func_name = sys._getframe().f_code.co_name
+        res, check = api_request([self.collection.query_iterator, batch_size, limit, expr, output_fields, partition_names,
+                                  timeout], **kwargs)
+        check_result = ResponseChecker(res, func_name, check_task, check_items, check,
+                                       batch_size=batch_size, limit=limit, expression=expr,
+                                       output_fields=output_fields, partition_names=partition_names,
+                                       timeout=timeout, **kwargs).run()
+        return res, check_result
+
+    @property
+    def partitions(self):
+        return self.collection.partitions
+
+    @trace()
+    def partition(self, partition_name, check_task=None, check_items=None):
+        func_name = sys._getframe().f_code.co_name
+        res, succ = api_request([self.collection.partition, partition_name])
+        check_result = ResponseChecker(res, func_name, check_task, check_items,
+                                       succ, partition_name=partition_name).run()
+        return res, check_result
+
+    @trace()
+    def has_partition(self, partition_name, check_task=None, check_items=None):
+        func_name = sys._getframe().f_code.co_name
+        res, succ = api_request([self.collection.has_partition, partition_name])
+        check_result = ResponseChecker(res, func_name, check_task, check_items,
+                                       succ, partition_name=partition_name).run()
+        return res, check_result
+
+    @trace()
+    def drop_partition(self, partition_name, check_task=None, check_items=None, **kwargs):
+        timeout = kwargs.get("timeout", TIMEOUT)
+        kwargs.update({"timeout": timeout})
+
+        func_name = sys._getframe().f_code.co_name
+        res, check = api_request([self.collection.drop_partition, partition_name], **kwargs)
+        check_result = ResponseChecker(res, func_name, check_task, check_items, check, partition_name=partition_name,
+                                       **kwargs).run()
+        return res, check_result
+
+    @trace()
+    def create_partition(self, partition_name, check_task=None, check_items=None, description=""):
+        func_name = sys._getframe().f_code.co_name
+        res, check = api_request([self.collection.create_partition, partition_name, description])
+        check_result = ResponseChecker(res, func_name, check_task, check_items, check,
+                                       partition_name=partition_name).run()
+        return res, check_result
+
+    @property
+    def indexes(self):
+        return self.collection.indexes
+
+    @trace()
+    def index(self, check_task=None, check_items=None, **kwargs):
+        func_name = sys._getframe().f_code.co_name
+        if "index_name" in kwargs:
+            index_name = kwargs.get('index_name')
+            kwargs.update({"index_name": index_name})
+        res, check = api_request([self.collection.index], **kwargs)
+        check_result = ResponseChecker(res, func_name, check_task, check_items, check, **kwargs).run()
+        return res, check_result
+
+    @trace()
+    def create_index(self, field_name, index_params=None, index_name=None, timeout=None, check_task=None, check_items=None, **kwargs):
+        timeout = 1200 if timeout is None else timeout
+        index_name = INDEX_NAME if index_name is None else index_name
+        index_name = kwargs.get("index_name", index_name)
+        kwargs.update({"index_name": index_name})
+        func_name = sys._getframe().f_code.co_name
+        res, check = api_request([self.collection.create_index, field_name, index_params, timeout], **kwargs)
+        check_result = ResponseChecker(res, func_name, check_task, check_items, check, **kwargs).run()
+        return res, check_result
+
+    @trace()
+    def has_index(self, index_name=None, check_task=None, check_items=None, **kwargs):
+        index_name = INDEX_NAME if index_name is None else index_name
+        index_name = kwargs.get("index_name", index_name)
+        kwargs.update({"index_name": index_name})
+
+        func_name = sys._getframe().f_code.co_name
+        res, check = api_request([self.collection.has_index], **kwargs)
+        check_result = ResponseChecker(res, func_name, check_task, check_items, check, **kwargs).run()
+        return res, check_result
+
+    @trace()
+    def drop_index(self, index_name=None, check_task=None, check_items=None, **kwargs):
+        timeout = kwargs.get("timeout", TIMEOUT)
+        index_name = INDEX_NAME if index_name is None else index_name
+        index_name = kwargs.get("index_name", index_name)
+        kwargs.update({"timeout": timeout, "index_name": index_name})
+
+        func_name = sys._getframe().f_code.co_name
+        res, check = api_request([self.collection.drop_index], **kwargs)
+        check_result = ResponseChecker(res, func_name, check_task, check_items, check, **kwargs).run()
+        return res, check_result
+
+    @trace()
+    def delete(self, expr, partition_name=None, timeout=None, check_task=None, check_items=None, **kwargs):
+        timeout = TIMEOUT if timeout is None else timeout
+        func_name = sys._getframe().f_code.co_name
+        res, check = api_request([self.collection.delete, expr, partition_name, timeout], **kwargs)
+        check_result = ResponseChecker(res, func_name, check_task, check_items, check, **kwargs).run()
+        return res, check_result
+
+    @trace()
+    def upsert(self, data, partition_name=None, timeout=None, check_task=None, check_items=None, **kwargs):
+        timeout = TIMEOUT if timeout is None else timeout
+        func_name = sys._getframe().f_code.co_name
+        res, check = api_request([self.collection.upsert, data, partition_name, timeout], **kwargs)
+        check_result = ResponseChecker(res, func_name, check_task, check_items, check, **kwargs).run()
+        return res, check_result
+
+    @trace()
+    def compact(self, is_clustering=False, timeout=None, check_task=None, check_items=None, **kwargs):
+        timeout = TIMEOUT if timeout is None else timeout
+        func_name = sys._getframe().f_code.co_name
+        res, check = api_request([self.collection.compact, is_clustering, timeout], **kwargs)
+        check_result = ResponseChecker(res, func_name, check_task, check_items, check, **kwargs).run()
+        return res, check_result
+
+    @trace()
+    def get_compaction_state(self, timeout=None, check_task=None, check_items=None, **kwargs):
+        timeout = TIMEOUT if timeout is None else timeout
+        func_name = sys._getframe().f_code.co_name
+        res, check = api_request([self.collection.get_compaction_state, timeout], **kwargs)
+        check_result = ResponseChecker(res, func_name, check_task, check_items, check, **kwargs).run()
+        return res, check_result
+
+    @trace()
+    def get_compaction_plans(self, timeout=None, check_task=None, check_items={}, **kwargs):
+        timeout = TIMEOUT if timeout is None else timeout
+        func_name = sys._getframe().f_code.co_name
+        res, check = api_request([self.collection.get_compaction_plans, timeout], **kwargs)
+        check_result = ResponseChecker(res, func_name, check_task, check_items, check, **kwargs).run()
+        return res, check_result
+
+    def wait_for_compaction_completed(self, timeout=None, **kwargs):
+        timeout = TIMEOUT * 3 if timeout is None else timeout
+        res = self.collection.wait_for_compaction_completed(timeout, **kwargs)
+        # log.debug(res)
+        return res
+
+    @trace()
+    def get_replicas(self, timeout=None, check_task=None, check_items=None, **kwargs):
+        timeout = TIMEOUT if timeout is None else timeout
+        func_name = sys._getframe().f_code.co_name
+        res, check = api_request([self.collection.get_replicas, timeout], **kwargs)
+        check_result = ResponseChecker(res, func_name, check_task, check_items, check, **kwargs).run()
+        return res, check_result
+
+    @trace()
+    def describe(self, timeout=None, check_task=None, check_items=None):
+        timeout = TIMEOUT if timeout is None else timeout
+        func_name = sys._getframe().f_code.co_name
+        res, check = api_request([self.collection.describe, timeout])
+        check_result = ResponseChecker(res, func_name, check_task, check_items, check).run()
+        return res, check_result
+
+    @trace()
+    def alter_index(self, index_name, extra_params={}, timeout=None,  check_task=None, check_items=None, **kwargs):
+        timeout = TIMEOUT if timeout is None else timeout
+        func_name = sys._getframe().f_code.co_name
+        res, check = api_request([self.collection.alter_index, index_name, extra_params, timeout], **kwargs)
+        check_result = ResponseChecker(res, func_name, check_task, check_items, check, **kwargs).run()
+        return res, check_result
+
+    @trace()
+    def set_properties(self, extra_params={}, timeout=None,  check_task=None, check_items=None, **kwargs):
+        timeout = TIMEOUT if timeout is None else timeout
+        func_name = sys._getframe().f_code.co_name
+        res, check = api_request([self.collection.set_properties, extra_params, timeout], **kwargs)
+        check_result = ResponseChecker(res, func_name, check_task, check_items, check, **kwargs).run()
+        return res, check_result

tests/python_client/base/connections_wrapper.py

@@ -0,0 +1,70 @@
+from pymilvus import Connections
+from pymilvus import DefaultConfig
+from pymilvus import MilvusClient
+import sys
+
+sys.path.append("..")
+from check.func_check import ResponseChecker
+from utils.api_request import api_request
+
+
+class ApiConnectionsWrapper:
+    def __init__(self):
+        self.connection = Connections()
+
+    def add_connection(self, check_task=None, check_items=None, **kwargs):
+        func_name = sys._getframe().f_code.co_name
+        response, is_succ = api_request([self.connection.add_connection], **kwargs)
+        check_result = ResponseChecker(response, func_name, check_task, check_items, is_succ, **kwargs).run()
+        return response, check_result
+
+    def disconnect(self, alias, check_task=None, check_items=None):
+        func_name = sys._getframe().f_code.co_name
+        response, is_succ = api_request([self.connection.disconnect, alias])
+        check_result = ResponseChecker(response, func_name, check_task, check_items, is_succ, alias=alias).run()
+        return response, check_result
+
+    def remove_connection(self, alias, check_task=None, check_items=None):
+        func_name = sys._getframe().f_code.co_name
+        response, is_succ = api_request([self.connection.remove_connection, alias])
+        check_result = ResponseChecker(response, func_name, check_task, check_items, is_succ, alias=alias).run()
+        return response, check_result
+
+    def connect(self, alias=DefaultConfig.DEFAULT_USING, user="", password="", db_name="default", token: str = "",
+                check_task=None, check_items=None, **kwargs):
+        func_name = sys._getframe().f_code.co_name
+        response, succ = api_request([self.connection.connect, alias, user, password, db_name, token], **kwargs)
+        check_result = ResponseChecker(response, func_name, check_task, check_items, succ, alias=alias, user=user,
+                                       password=password, db_name=db_name, token=token, **kwargs).run()
+        return response, check_result
+
+    def has_connection(self, alias=DefaultConfig.DEFAULT_USING, check_task=None, check_items=None):
+        func_name = sys._getframe().f_code.co_name
+        response, succ = api_request([self.connection.has_connection, alias])
+        check_result = ResponseChecker(response, func_name, check_task, check_items, succ, alias=alias).run()
+        return response, check_result
+
+    #  def get_connection(self, alias=DefaultConfig.DEFAULT_USING, check_task=None, check_items=None):
+    #      func_name = sys._getframe().f_code.co_name
+    #      response, is_succ = api_request([self.connection.get_connection, alias])
+    #      check_result = ResponseChecker(response, func_name, check_task, check_items, is_succ, alias=alias).run()
+    #      return response, check_result
+
+    def list_connections(self, check_task=None, check_items=None):
+        func_name = sys._getframe().f_code.co_name
+        response, is_succ = api_request([self.connection.list_connections])
+        check_result = ResponseChecker(response, func_name, check_task, check_items, is_succ).run()
+        return response, check_result
+
+    def get_connection_addr(self, alias, check_task=None, check_items=None):
+        func_name = sys._getframe().f_code.co_name
+        response, is_succ = api_request([self.connection.get_connection_addr, alias])
+        check_result = ResponseChecker(response, func_name, check_task, check_items, is_succ, alias=alias).run()
+        return response, check_result
+    
+    # high level api
+    def MilvusClient(self, check_task=None, check_items=None, **kwargs):
+        func_name = sys._getframe().f_code.co_name
+        response, succ = api_request([MilvusClient], **kwargs)
+        check_result = ResponseChecker(response, func_name, check_task, check_items, succ, **kwargs).run()
+        return response, check_result

tests/python_client/base/database_wrapper.py

@@ -0,0 +1,47 @@
+import sys
+
+from pymilvus import db
+
+from utils.api_request import api_request
+from check.func_check import ResponseChecker
+
+
+class ApiDatabaseWrapper:
+    """ wrapper of database """
+    database = db
+
+    def create_database(self, db_name, using="default", timeout=None, check_task=None, check_items=None):
+        func_name = sys._getframe().f_code.co_name
+        response, is_succ = api_request([self.database.create_database, db_name, using, timeout])
+        check_result = ResponseChecker(response, func_name, check_task, check_items, is_succ).run()
+        return response, check_result
+
+    def using_database(self, db_name, using="default", check_task=None, check_items=None):
+        func_name = sys._getframe().f_code.co_name
+        response, is_succ = api_request([self.database.using_database, db_name, using])
+        check_result = ResponseChecker(response, func_name, check_task, check_items, is_succ).run()
+        return response, check_result
+
+    def drop_database(self, db_name, using="default", timeout=None, check_task=None, check_items=None):
+        func_name = sys._getframe().f_code.co_name
+        response, is_succ = api_request([self.database.drop_database, db_name, using, timeout])
+        check_result = ResponseChecker(response, func_name, check_task, check_items, is_succ).run()
+        return response, check_result
+
+    def list_database(self, using="default", timeout=None, check_task=None, check_items=None):
+        func_name = sys._getframe().f_code.co_name
+        response, is_succ = api_request([self.database.list_database, using, timeout])
+        check_result = ResponseChecker(response, func_name, check_task, check_items, is_succ).run()
+        return response, check_result
+
+    def set_properties(self, db_name: str, properties: dict, using="default", timeout=None, check_task=None, check_items=None):
+        func_name = sys._getframe().f_code.co_name
+        response, is_succ = api_request([self.database.set_properties, db_name, properties, using, timeout])
+        check_result = ResponseChecker(response, func_name, check_task, check_items, is_succ).run()
+        return response, check_result
+
+    def describe_database(self, db_name: str, using="default", timeout=None, check_task=None, check_items=None):
+        func_name = sys._getframe().f_code.co_name
+        response, is_succ = api_request([self.database.describe_database, db_name, using, timeout])
+        check_result = ResponseChecker(response, func_name, check_task, check_items, is_succ).run()
+        return response, check_result

tests/python_client/base/index_wrapper.py

@@ -0,0 +1,53 @@
+import sys
+from pymilvus import Index
+
+sys.path.append("..")
+from check.func_check import ResponseChecker
+from utils.api_request import api_request
+
+TIMEOUT = 20
+INDEX_NAME = ""
+
+
+class ApiIndexWrapper:
+    index = None
+
+    def init_index(self, collection, field_name, index_params, index_name=None, check_task=None, check_items=None,
+                   **kwargs):
+        disktimeout = 600
+        timeout = kwargs.get("timeout", disktimeout * 2)
+        index_name = INDEX_NAME if index_name is None else index_name
+        index_name = kwargs.get("index_name", index_name)
+        kwargs.update({"timeout": timeout, "index_name": index_name})
+
+        """ In order to distinguish the same name of index """
+        func_name = sys._getframe().f_code.co_name
+        res, is_succ = api_request([Index, collection, field_name, index_params], **kwargs)
+        self.index = res if is_succ is True else None
+        check_result = ResponseChecker(res, func_name, check_task, check_items, is_succ,
+                                       collection=collection, field_name=field_name,
+                                       index_params=index_params, **kwargs).run()
+        return res, check_result
+
+    def drop(self, index_name=None, check_task=None, check_items=None, **kwargs):
+        timeout = kwargs.get("timeout", TIMEOUT)
+        index_name = INDEX_NAME if index_name is None else index_name
+        index_name = kwargs.get("index_name", index_name)
+        kwargs.update({"timeout": timeout, "index_name": index_name})
+
+        func_name = sys._getframe().f_code.co_name
+        res, is_succ = api_request([self.index.drop], **kwargs)
+        check_result = ResponseChecker(res, func_name, check_task, check_items, is_succ, **kwargs).run()
+        return res, check_result
+
+    @property
+    def params(self):
+        return self.index.params
+
+    @property
+    def collection_name(self):
+        return self.index.collection_name
+
+    @property
+    def field_name(self):
+        return self.index.field_name

tests/python_client/base/partition_wrapper.py

@@ -0,0 +1,159 @@
+import sys
+from numpy import NaN
+
+from pymilvus import Partition
+
+sys.path.append("..")
+from check.func_check import ResponseChecker
+from utils.api_request import api_request
+from common.common_func import param_info
+
+
+TIMEOUT = 180
+
+
+class ApiPartitionWrapper:
+    partition = None
+
+    def init_partition(self, collection, name, description="",
+                       check_task=None, check_items=None, **kwargs):
+        """ In order to distinguish the same name of partition """
+        func_name = sys._getframe().f_code.co_name
+        response, is_succ = api_request([Partition, collection, name, description], **kwargs)
+        self.partition = response if is_succ is True else None
+        check_result = ResponseChecker(response, func_name, check_task, check_items, is_succ,
+                                       **kwargs).run()
+        return response, check_result
+
+    @property
+    def description(self):
+        return self.partition.description if self.partition else None
+
+    @property
+    def name(self):
+        return self.partition.name if self.partition else None
+
+    @property
+    def is_empty(self):
+        self.flush()
+        return self.partition.is_empty if self.partition else None
+
+    @property
+    def is_empty_without_flush(self):
+        return self.partition.is_empty if self.partition else None
+
+    @property
+    def num_entities(self):
+        self.flush()
+        return self.partition.num_entities if self.partition else None
+
+    @property
+    def num_entities_without_flush(self):
+        return self.partition.num_entities if self.partition else None
+
+    def drop(self, check_task=None, check_items=None, **kwargs):
+        timeout = kwargs.get("timeout", TIMEOUT)
+        kwargs.update({"timeout": timeout})
+
+        func_name = sys._getframe().f_code.co_name
+        res, succ = api_request([self.partition.drop], **kwargs)
+        check_result = ResponseChecker(res, func_name,
+                                       check_task, check_items, succ, **kwargs).run()
+        return res, check_result
+
+    def load(self, replica_number=NaN, timeout=None, check_task=None, check_items=None, **kwargs):
+        timeout = TIMEOUT if timeout is None else timeout
+        replica_number = param_info.param_replica_num if replica_number is NaN else replica_number
+
+        func_name = sys._getframe().f_code.co_name
+        res, succ = api_request([self.partition.load, replica_number, timeout], **kwargs)
+        check_result = ResponseChecker(res, func_name, check_task,
+                                       check_items, is_succ=succ,
+                                       **kwargs).run()
+        return res, check_result
+
+    def release(self, check_task=None, check_items=None, **kwargs):
+        timeout = kwargs.get("timeout", TIMEOUT)
+        kwargs.update({"timeout": timeout})
+
+        func_name = sys._getframe().f_code.co_name
+        res, succ = api_request([self.partition.release], **kwargs)
+        check_result = ResponseChecker(res, func_name, check_task,
+                                       check_items, is_succ=succ,
+                                       **kwargs).run()
+        return res, check_result
+
+    def flush(self, check_task=None, check_items=None, **kwargs):
+        timeout = kwargs.get("timeout", TIMEOUT)
+        kwargs.update({"timeout": timeout})
+
+        func_name = sys._getframe().f_code.co_name
+        res, succ = api_request([self.partition.flush], **kwargs)
+        check_result = ResponseChecker(res, func_name, check_task,
+                                       check_items, is_succ=succ,
+                                       **kwargs).run()
+        return res, check_result
+
+    def insert(self, data, check_task=None, check_items=None, **kwargs):
+        timeout = kwargs.get("timeout", TIMEOUT)
+        kwargs.update({"timeout": timeout})
+
+        func_name = sys._getframe().f_code.co_name
+        res, succ = api_request([self.partition.insert, data], **kwargs)
+        check_result = ResponseChecker(res, func_name, check_task,
+                                       check_items, is_succ=succ, data=data,
+                                       **kwargs).run()
+        return res, check_result
+
+    def search(self, data, anns_field, params, limit, expr=None, output_fields=None,
+               check_task=None, check_items=None, **kwargs):
+        timeout = kwargs.get("timeout", TIMEOUT)
+        kwargs.update({"timeout": timeout})
+
+        func_name = sys._getframe().f_code.co_name
+        res, succ = api_request([self.partition.search, data, anns_field, params,
+                                 limit, expr, output_fields], **kwargs)
+        check_result = ResponseChecker(res, func_name, check_task, check_items,
+                                       is_succ=succ, data=data, anns_field=anns_field,
+                                       params=params, limit=limit, expr=expr,
+                                       output_fields=output_fields, **kwargs).run()
+        return res, check_result
+
+    def query(self, expr, output_fields=None, timeout=None, check_task=None, check_items=None, **kwargs):
+        timeout = TIMEOUT if timeout is None else timeout
+
+        func_name = sys._getframe().f_code.co_name
+        res, check = api_request([self.partition.query, expr, output_fields, timeout], **kwargs)
+        check_result = ResponseChecker(res, func_name, check_task, check_items, check,
+                                       expression=expr, output_fields=output_fields,
+                                       timeout=timeout, **kwargs).run()
+        return res, check_result
+
+    def delete(self, expr, check_task=None, check_items=None, **kwargs):
+        timeout = kwargs.get("timeout", TIMEOUT)
+        kwargs.update({"timeout": timeout})
+
+        func_name = sys._getframe().f_code.co_name
+        res, succ = api_request([self.partition.delete, expr], **kwargs)
+        check_result = ResponseChecker(res, func_name, check_task,
+                                       check_items, is_succ=succ, expr=expr,
+                                       **kwargs).run()
+        return res, check_result
+
+    def upsert(self, data, check_task=None, check_items=None, **kwargs):
+        timeout = kwargs.get("timeout", TIMEOUT)
+        kwargs.update({"timeout": timeout})
+
+        func_name = sys._getframe().f_code.co_name
+        res, succ = api_request([self.partition.upsert, data], **kwargs)
+        check_result = ResponseChecker(res, func_name, check_task,
+                                       check_items, is_succ=succ, data=data,
+                                       **kwargs).run()
+        return res, check_result
+
+    def get_replicas(self, timeout=None, check_task=None, check_items=None, **kwargs):
+        timeout = TIMEOUT if timeout is None else timeout
+        func_name = sys._getframe().f_code.co_name
+        res, check = api_request([self.partition.get_replicas, timeout], **kwargs)
+        check_result = ResponseChecker(res, func_name, check_task, check_items, check, **kwargs).run()
+        return res, check_result

tests/python_client/base/schema_wrapper.py

@@ -0,0 +1,83 @@
+import sys
+
+sys.path.append("..")
+from check.func_check import ResponseChecker
+from utils.api_request import api_request
+from pymilvus import CollectionSchema, FieldSchema
+
+
+class ApiCollectionSchemaWrapper:
+    collection_schema = None
+
+    def init_collection_schema(self, fields, description="", check_task=None, check_items=None, **kwargs):
+        """In order to distinguish the same name of CollectionSchema"""
+        func_name = sys._getframe().f_code.co_name
+        response, is_succ = api_request([CollectionSchema, fields, description], **kwargs)
+        self.collection_schema = response if is_succ else None
+        check_result = ResponseChecker(response, func_name, check_task, check_items, is_succ=is_succ, fields=fields,
+                                       description=description, **kwargs).run()
+        return response, check_result
+
+    @property
+    def primary_field(self):
+        return self.collection_schema.primary_field if self.collection_schema else None
+
+    @property
+    def partition_key_field(self):
+        return self.collection_schema.partition_key_field if self.collection_schema else None
+
+    @property
+    def fields(self):
+        return self.collection_schema.fields if self.collection_schema else None
+
+    @property
+    def description(self):
+        return self.collection_schema.description if self.collection_schema else None
+
+    @property
+    def auto_id(self):
+        return self.collection_schema.auto_id if self.collection_schema else None
+
+    @property
+    def enable_dynamic_field(self):
+        return self.collection_schema.enable_dynamic_field if self.collection_schema else None
+
+    @property
+    def to_dict(self):
+        return self.collection_schema.to_dict if self.collection_schema else None
+
+    @property
+    def verify(self):
+        return self.collection_schema.verify if self.collection_schema else None
+
+    def add_field(self, field_name, datatype, check_task=None, check_items=None, **kwargs):
+        func_name = sys._getframe().f_code.co_name
+        response, is_succ = api_request([self.collection_schema.add_field, field_name, datatype], **kwargs)
+        check_result = ResponseChecker(response, func_name, check_task, check_items,
+                                       field_name=field_name, datatype=datatype, **kwargs).run()
+        return response, check_result
+
+
+class ApiFieldSchemaWrapper:
+    field_schema = None
+
+    def init_field_schema(self, name, dtype, description="", check_task=None, check_items=None, **kwargs):
+        """In order to distinguish the same name of FieldSchema"""
+        func_name = sys._getframe().f_code.co_name
+        response, is_succ = api_request([FieldSchema, name, dtype, description], **kwargs)
+        self.field_schema = response if is_succ else None
+        check_result = ResponseChecker(response, func_name, check_task, check_items, is_succ, name=name, dtype=dtype,
+                                       description=description, **kwargs).run()
+        return response, check_result
+
+    @property
+    def description(self):
+        return self.field_schema.description if self.field_schema else None
+
+    @property
+    def params(self):
+        return self.field_schema.params if self.field_schema else None
+
+    @property
+    def dtype(self):
+        return self.field_schema.dtype if self.field_schema else None

tests/python_client/base/utility_wrapper.py

@@ -0,0 +1,602 @@
+from datetime import datetime
+import time
+from pymilvus import utility
+import sys
+
+sys.path.append("..")
+from check.func_check import ResponseChecker
+from utils.api_request import api_request
+from pymilvus import BulkInsertState
+from pymilvus.orm.role import Role
+from utils.util_log import test_log as log
+
+TIMEOUT = 20
+
+
+class ApiUtilityWrapper:
+    """ Method of encapsulating utility files """
+
+    ut = utility
+    role = None
+
+    def do_bulk_insert(self, collection_name, files="", partition_name=None, timeout=None,
+                       using="default", check_task=None, check_items=None, **kwargs):
+        working_tasks = self.get_bulk_insert_working_list()
+        log.info(f"before bulk load, there are {len(working_tasks)} working tasks")
+        log.info(f"files to load: {files}")
+        func_name = sys._getframe().f_code.co_name
+        res, is_succ = api_request([self.ut.do_bulk_insert, collection_name,
+                                    files, partition_name, timeout, using], **kwargs)
+        check_result = ResponseChecker(res, func_name, check_task, check_items, is_succ,
+                                       collection_name=collection_name, using=using).run()
+        time.sleep(1)
+        working_tasks = self.get_bulk_insert_working_list()
+        log.info(f"after bulk load, there are {len(working_tasks)} working tasks")
+        return res, check_result
+
+    def get_bulk_insert_state(self, task_id, timeout=None, using="default", check_task=None, check_items=None,
+                              **kwargs):
+        func_name = sys._getframe().f_code.co_name
+        res, is_succ = api_request([self.ut.get_bulk_insert_state, task_id, timeout, using], **kwargs)
+        check_result = ResponseChecker(res, func_name, check_task, check_items, is_succ,
+                                       task_id=task_id, using=using).run()
+        return res, check_result
+
+    def list_bulk_insert_tasks(self, limit=0, collection_name=None, timeout=None, using="default", check_task=None,
+                               check_items=None, **kwargs):
+        func_name = sys._getframe().f_code.co_name
+        res, is_succ = api_request([self.ut.list_bulk_insert_tasks, limit, collection_name, timeout, using], **kwargs)
+        check_result = ResponseChecker(res, func_name, check_task, check_items, is_succ,
+                                       limit=limit, collection_name=collection_name, using=using).run()
+        return res, check_result
+
+    def get_bulk_insert_pending_list(self):
+        tasks = {}
+        for task in self.ut.list_bulk_insert_tasks():
+            if task.state == BulkInsertState.ImportPending:
+                tasks[task.task_id] = task
+        return tasks
+
+    def get_bulk_insert_working_list(self):
+        tasks = {}
+        for task in self.ut.list_bulk_insert_tasks():
+            if task.state in [BulkInsertState.ImportStarted]:
+                tasks[task.task_id] = task
+        return tasks
+
+    def list_all_bulk_insert_tasks(self, limit=0):
+        tasks, _ = self.list_bulk_insert_tasks(limit=limit)
+        pending = 0
+        started = 0
+        persisted = 0
+        completed = 0
+        failed = 0
+        failed_and_cleaned = 0
+        unknown = 0
+        for task in tasks:
+            print(task)
+            if task.state == BulkInsertState.ImportPending:
+                pending = pending + 1
+            elif task.state == BulkInsertState.ImportStarted:
+                started = started + 1
+            elif task.state == BulkInsertState.ImportPersisted:
+                persisted = persisted + 1
+            elif task.state == BulkInsertState.ImportCompleted:
+                completed = completed + 1
+            elif task.state == BulkInsertState.ImportFailed:
+                failed = failed + 1
+            elif task.state == BulkInsertState.ImportFailedAndCleaned:
+                failed_and_cleaned = failed_and_cleaned + 1
+            else:
+                unknown = unknown + 1
+
+        log.info("There are", len(tasks), "bulkload tasks.", pending, "pending,", started, "started,", persisted,
+                 "persisted,", completed, "completed,", failed, "failed", failed_and_cleaned, "failed_and_cleaned",
+                 unknown, "unknown")
+
+    def wait_for_bulk_insert_tasks_completed(self, task_ids, target_state=BulkInsertState.ImportCompleted,
+                                             timeout=None, using="default", **kwargs):
+        tasks_state_distribution = {
+            "success": set(),
+            "failed": set(),
+            "in_progress": set()
+        }
+        tasks_state = {}
+        if timeout is not None:
+            task_timeout = timeout
+        else:
+            task_timeout = TIMEOUT
+        start = time.time()
+        end = time.time()
+        log.info(f"wait bulk load timeout is {task_timeout}")
+        pending_tasks = self.get_bulk_insert_pending_list()
+        log.info(f"before waiting, there are {len(pending_tasks)} pending tasks")
+        while len(tasks_state_distribution["success"]) + len(tasks_state_distribution["failed"]) < len(
+                task_ids) and end - start <= task_timeout:
+            time.sleep(2)
+
+            for task_id in task_ids:
+                if task_id in tasks_state_distribution["success"] or task_id in tasks_state_distribution["failed"]:
+                    continue
+                else:
+                    state, _ = self.get_bulk_insert_state(task_id, task_timeout, using, **kwargs)
+                    tasks_state[task_id] = state
+
+                    if target_state == BulkInsertState.ImportPersisted:
+                        if state.state in [BulkInsertState.ImportPersisted, BulkInsertState.ImportCompleted]:
+                            if task_id in tasks_state_distribution["in_progress"]:
+                                tasks_state_distribution["in_progress"].remove(task_id)
+                            tasks_state_distribution["success"].add(task_id)
+                        elif state.state in [BulkInsertState.ImportPending, BulkInsertState.ImportStarted]:
+                            tasks_state_distribution["in_progress"].add(task_id)
+                        else:
+                            tasks_state_distribution["failed"].add(task_id)
+
+                    if target_state == BulkInsertState.ImportCompleted:
+                        if state.state in [BulkInsertState.ImportCompleted]:
+                            if task_id in tasks_state_distribution["in_progress"]:
+                                tasks_state_distribution["in_progress"].remove(task_id)
+                            tasks_state_distribution["success"].add(task_id)
+                        elif state.state in [BulkInsertState.ImportPending, BulkInsertState.ImportStarted,
+                                             BulkInsertState.ImportPersisted]:
+                            tasks_state_distribution["in_progress"].add(task_id)
+                        else:
+                            tasks_state_distribution["failed"].add(task_id)
+
+            end = time.time()
+        pending_tasks = self.get_bulk_insert_pending_list()
+        log.info(f"after waiting, there are {len(pending_tasks)} pending tasks")
+        log.info(f"task state distribution: {tasks_state_distribution}")
+        log.info(tasks_state)
+        if len(tasks_state_distribution["success"]) == len(task_ids):
+            log.info(f"wait for bulk load tasks completed successfully, cost time: {end - start}")
+            return True, tasks_state
+        else:
+            log.info(f"wait for bulk load tasks completed failed, cost time: {end - start}")
+            return False, tasks_state
+
+    def wait_all_pending_tasks_finished(self):
+        task_states_map = {}
+        all_tasks, _ = self.list_bulk_insert_tasks()
+        # log.info(f"all tasks: {all_tasks}")
+        for task in all_tasks:
+            if task.state in [BulkInsertState.ImportStarted, BulkInsertState.ImportPersisted]:
+                task_states_map[task.task_id] = task.state
+
+        log.info(f"current tasks states: {task_states_map}")
+        pending_tasks = self.get_bulk_insert_pending_list()
+        working_tasks = self.get_bulk_insert_working_list()
+        log.info(
+            f"in the start, there are {len(working_tasks)} working tasks, {working_tasks} {len(pending_tasks)} pending tasks, {pending_tasks}")
+        time_cnt = 0
+        pending_task_ids = set()
+        while len(pending_tasks) > 0:
+            time.sleep(5)
+            time_cnt += 5
+            pending_tasks = self.get_bulk_insert_pending_list()
+            working_tasks = self.get_bulk_insert_working_list()
+            cur_pending_task_ids = []
+            for task_id in pending_tasks.keys():
+                cur_pending_task_ids.append(task_id)
+                pending_task_ids.add(task_id)
+            log.info(
+                f"after {time_cnt}, there are {len(working_tasks)} working tasks, {len(pending_tasks)} pending tasks")
+            log.debug(f"total pending tasks: {pending_task_ids} current pending tasks: {cur_pending_task_ids}")
+        log.info(f"after {time_cnt}, all pending tasks are finished")
+        all_tasks, _ = self.list_bulk_insert_tasks()
+        for task in all_tasks:
+            if task.task_id in pending_task_ids:
+                log.info(f"task {task.task_id} state transfer from pending to {task.state_name}")
+
+    def wait_index_build_completed(self, collection_name, timeout=None):
+        start = time.time()
+        if timeout is not None:
+            task_timeout = timeout
+        else:
+            task_timeout = TIMEOUT
+        end = time.time()
+        while end - start <= task_timeout:
+            time.sleep(0.5)
+            index_states, _ = self.index_building_progress(collection_name)
+            log.debug(f"index states: {index_states}")
+            if index_states["total_rows"] == index_states["indexed_rows"]:
+                log.info(f"index build completed")
+                return True
+            end = time.time()
+        log.info(f"index build timeout")
+        return False
+
+    def get_query_segment_info(self, collection_name, timeout=None, using="default", check_task=None, check_items=None):
+        timeout = TIMEOUT if timeout is None else timeout
+        func_name = sys._getframe().f_code.co_name
+        res, is_succ = api_request([self.ut.get_query_segment_info, collection_name, timeout, using])
+        check_result = ResponseChecker(res, func_name, check_task, check_items, is_succ,
+                                       collection_name=collection_name, timeout=timeout, using=using).run()
+        return res, check_result
+
+    def loading_progress(self, collection_name, partition_names=None,
+                         using="default", check_task=None, check_items=None):
+        func_name = sys._getframe().f_code.co_name
+        res, is_succ = api_request([self.ut.loading_progress, collection_name, partition_names, using])
+        check_result = ResponseChecker(res, func_name, check_task,
+                                       check_items, is_succ, collection_name=collection_name,
+                                       partition_names=partition_names, using=using).run()
+        return res, check_result
+
+    def load_state(self, collection_name, partition_names=None, using="default", check_task=None, check_items=None):
+        func_name = sys._getframe().f_code.co_name
+        res, is_succ = api_request([self.ut.load_state, collection_name, partition_names, using])
+        check_result = ResponseChecker(res, func_name, check_task, check_items, is_succ,
+                                       collection_name=collection_name, partition_names=partition_names,
+                                       using=using).run()
+        return res, check_result
+
+    def wait_for_loading_complete(self, collection_name, partition_names=None, timeout=None, using="default",
+                                  check_task=None, check_items=None):
+        timeout = TIMEOUT if timeout is None else timeout
+
+        func_name = sys._getframe().f_code.co_name
+        res, is_succ = api_request([self.ut.wait_for_loading_complete, collection_name,
+                                    partition_names, timeout, using])
+        check_result = ResponseChecker(res, func_name, check_task, check_items, is_succ,
+                                       collection_name=collection_name, partition_names=partition_names,
+                                       timeout=timeout, using=using).run()
+        return res, check_result
+
+    def index_building_progress(self, collection_name, index_name="", using="default",
+                                check_task=None, check_items=None):
+        func_name = sys._getframe().f_code.co_name
+        res, is_succ = api_request([self.ut.index_building_progress, collection_name, index_name, using])
+        check_result = ResponseChecker(res, func_name, check_task, check_items, is_succ,
+                                       collection_name=collection_name, index_name=index_name,
+                                       using=using).run()
+        return res, check_result
+
+    def wait_for_index_building_complete(self, collection_name, index_name="", timeout=None, using="default",
+                                         check_task=None, check_items=None):
+        timeout = TIMEOUT if timeout is None else timeout
+
+        func_name = sys._getframe().f_code.co_name
+        res, is_succ = api_request([self.ut.wait_for_index_building_complete, collection_name,
+                                    index_name, timeout, using])
+        check_result = ResponseChecker(res, func_name, check_task, check_items, is_succ,
+                                       collection_name=collection_name, index_name=index_name,
+                                       timeout=timeout, using=using).run()
+        return res, check_result
+
+    def has_collection(self, collection_name, using="default", check_task=None, check_items=None):
+        func_name = sys._getframe().f_code.co_name
+        res, is_succ = api_request([self.ut.has_collection, collection_name, using])
+        check_result = ResponseChecker(res, func_name, check_task, check_items, is_succ,
+                                       collection_name=collection_name, using=using).run()
+        return res, check_result
+
+    def has_partition(self, collection_name, partition_name, using="default",
+                      check_task=None, check_items=None):
+        func_name = sys._getframe().f_code.co_name
+        res, is_succ = api_request([self.ut.has_partition, collection_name, partition_name, using])
+        check_result = ResponseChecker(res, func_name, check_task, check_items, is_succ,
+                                       collection_name=collection_name,
+                                       partition_name=partition_name, using=using).run()
+        return res, check_result
+
+    def drop_collection(self, collection_name, timeout=None, using="default", check_task=None, check_items=None):
+        func_name = sys._getframe().f_code.co_name
+        res, is_succ = api_request([self.ut.drop_collection, collection_name, timeout, using])
+        check_result = ResponseChecker(res, func_name, check_task, check_items, is_succ,
+                                       collection_name=collection_name,
+                                       timeout=timeout, using=using).run()
+        return res, check_result
+
+    def list_collections(self, timeout=None, using="default", check_task=None, check_items=None):
+        timeout = TIMEOUT if timeout is None else timeout
+
+        func_name = sys._getframe().f_code.co_name
+        res, is_succ = api_request([self.ut.list_collections, timeout, using])
+        check_result = ResponseChecker(res, func_name, check_task, check_items, is_succ,
+                                       timeout=timeout, using=using).run()
+        return res, check_result
+
+    def calc_distance(self, vectors_left, vectors_right, params=None, timeout=None,
+                      using="default", check_task=None, check_items=None):
+        timeout = TIMEOUT if timeout is None else timeout
+
+        func_name = sys._getframe().f_code.co_name
+        res, is_succ = api_request([self.ut.calc_distance, vectors_left, vectors_right,
+                                    params, timeout, using])
+        check_result = ResponseChecker(res, func_name, check_task, check_items, is_succ,
+                                       timeout=timeout, using=using).run()
+        return res, check_result
+
+    def load_balance(self, collection_name, src_node_id, dst_node_ids, sealed_segment_ids, timeout=None,
+                     using="default", check_task=None, check_items=None):
+        timeout = TIMEOUT if timeout is None else timeout
+
+        func_name = sys._getframe().f_code.co_name
+        res, is_succ = api_request([self.ut.load_balance, collection_name, src_node_id, dst_node_ids,
+                                    sealed_segment_ids, timeout, using])
+        check_result = ResponseChecker(res, func_name, check_task, check_items, is_succ,
+                                       timeout=timeout, using=using).run()
+        return res, check_result
+
+    def create_alias(self, collection_name, alias, timeout=None, using="default", check_task=None, check_items=None):
+        timeout = TIMEOUT if timeout is None else timeout
+        func_name = sys._getframe().f_code.co_name
+        res, is_succ = api_request([self.ut.create_alias, collection_name, alias, timeout, using])
+        check_result = ResponseChecker(res, func_name, check_task, check_items, is_succ,
+                                       timeout=timeout, using=using).run()
+        return res, check_result
+
+    def drop_alias(self, alias, timeout=None, using="default", check_task=None, check_items=None):
+        timeout = TIMEOUT if timeout is None else timeout
+        func_name = sys._getframe().f_code.co_name
+        res, is_succ = api_request([self.ut.drop_alias, alias, timeout, using])
+        check_result = ResponseChecker(res, func_name, check_task, check_items, is_succ,
+                                       timeout=timeout, using=using).run()
+        return res, check_result
+
+    def alter_alias(self, collection_name, alias, timeout=None, using="default", check_task=None, check_items=None):
+        timeout = TIMEOUT if timeout is None else timeout
+        func_name = sys._getframe().f_code.co_name
+        res, is_succ = api_request([self.ut.alter_alias, collection_name, alias, timeout, using])
+        check_result = ResponseChecker(res, func_name, check_task, check_items, is_succ,
+                                       timeout=timeout, using=using).run()
+        return res, check_result
+
+    def list_aliases(self, collection_name, timeout=None, using="default", check_task=None, check_items=None):
+        timeout = TIMEOUT if timeout is None else timeout
+        func_name = sys._getframe().f_code.co_name
+        res, is_succ = api_request([self.ut.list_aliases, collection_name, timeout, using])
+        check_result = ResponseChecker(res, func_name, check_task, check_items, is_succ,
+                                       timeout=timeout, using=using).run()
+        return res, check_result
+
+    def mkts_from_datetime(self, d_time=None, milliseconds=0., delta=None):
+        d_time = datetime.now() if d_time is None else d_time
+        res, _ = api_request([self.ut.mkts_from_datetime, d_time, milliseconds, delta])
+        return res
+
+    def mkts_from_hybridts(self, hybridts, milliseconds=0., delta=None):
+        res, _ = api_request([self.ut.mkts_from_hybridts, hybridts, milliseconds, delta])
+        return res
+
+    def create_user(self, user, password, using="default", check_task=None, check_items=None):
+        func_name = sys._getframe().f_code.co_name
+        res, is_succ = api_request([self.ut.create_user, user, password, using])
+        check_result = ResponseChecker(res, func_name, check_task, check_items, is_succ, using=using).run()
+        return res, check_result
+
+    def list_usernames(self, using="default", check_task=None, check_items=None):
+        func_name = sys._getframe().f_code.co_name
+        res, is_succ = api_request([self.ut.list_usernames, using])
+        check_result = ResponseChecker(res, func_name, check_task, check_items, is_succ,
+                                       using=using).run()
+        return res, check_result
+
+    def reset_password(self, user, old_password, new_password, check_task=None, check_items=None):
+        func_name = sys._getframe().f_code.co_name
+        res, is_succ = api_request([self.ut.reset_password, user, old_password, new_password])
+        check_result = ResponseChecker(res, func_name, check_task, check_items, is_succ).run()
+        return res, check_result
+
+    def update_password(self, user, old_password, new_password, check_task=None, check_items=None):
+        func_name = sys._getframe().f_code.co_name
+        res, is_succ = api_request([self.ut.update_password, user, old_password, new_password])
+        check_result = ResponseChecker(res, func_name, check_task, check_items, is_succ).run()
+        return res, check_result
+
+    def delete_user(self, user, using="default", check_task=None, check_items=None):
+        func_name = sys._getframe().f_code.co_name
+        res, is_succ = api_request([self.ut.delete_user, user, using])
+        check_result = ResponseChecker(res, func_name, check_task, check_items, is_succ,
+                                       using=using).run()
+        return res, check_result
+
+    def list_roles(self, include_user_info: bool, using="default", check_task=None, check_items=None):
+        func_name = sys._getframe().f_code.co_name
+        res, is_succ = api_request([self.ut.list_roles, include_user_info, using])
+        check_result = ResponseChecker(res, func_name, check_task, check_items, is_succ, using=using).run()
+        return res, check_result
+
+    def list_user(self, username: str, include_role_info: bool, using="default", check_task=None, check_items=None):
+        func_name = sys._getframe().f_code.co_name
+        res, is_succ = api_request([self.ut.list_user, username, include_role_info, using])
+        check_result = ResponseChecker(res, func_name, check_task, check_items, is_succ, using=using).run()
+        return res, check_result
+
+    def list_users(self, include_role_info: bool, using="default", check_task=None, check_items=None):
+        func_name = sys._getframe().f_code.co_name
+        res, is_succ = api_request([self.ut.list_users, include_role_info, using])
+        check_result = ResponseChecker(res, func_name, check_task, check_items, is_succ, using=using).run()
+        return res, check_result
+
+    def init_role(self, name, using="default", check_task=None, check_items=None, **kwargs):
+        func_name = sys._getframe().f_code.co_name
+        res, is_succ = api_request([Role, name, using], **kwargs)
+        self.role = res if is_succ else None
+        check_result = ResponseChecker(res, func_name, check_task, check_items, is_succ,
+                                       name=name, **kwargs).run()
+        return res, check_result
+
+    def create_role(self, check_task=None, check_items=None, **kwargs):
+        func_name = sys._getframe().f_code.co_name
+        res, is_succ = api_request([self.role.create], **kwargs)
+        check_result = ResponseChecker(res, func_name, check_task, check_items, is_succ,
+                                       **kwargs).run()
+        return res, check_result
+
+    def role_drop(self, check_task=None, check_items=None, **kwargs):
+        func_name = sys._getframe().f_code.co_name
+        res, check = api_request([self.role.drop], **kwargs)
+        check_result = ResponseChecker(res, func_name, check_task, check_items, check, **kwargs).run()
+        return res, check_result
+
+    def role_is_exist(self, check_task=None, check_items=None, **kwargs):
+        func_name = sys._getframe().f_code.co_name
+        res, check = api_request([self.role.is_exist], **kwargs)
+        check_result = ResponseChecker(res, func_name, check_task, check_items, check, **kwargs).run()
+        return res, check_result
+
+    def role_add_user(self, username: str, check_task=None, check_items=None, **kwargs):
+        func_name = sys._getframe().f_code.co_name
+        res, check = api_request([self.role.add_user, username], **kwargs)
+        check_result = ResponseChecker(res, func_name, check_task, check_items, check, **kwargs).run()
+        return res, check_result
+
+    def role_remove_user(self, username: str, check_task=None, check_items=None, **kwargs):
+        func_name = sys._getframe().f_code.co_name
+        res, check = api_request([self.role.remove_user, username], **kwargs)
+        check_result = ResponseChecker(res, func_name, check_task, check_items, check, **kwargs).run()
+        return res, check_result
+
+    def role_get_users(self, check_task=None, check_items=None, **kwargs):
+        func_name = sys._getframe().f_code.co_name
+        res, check = api_request([self.role.get_users], **kwargs)
+        check_result = ResponseChecker(res, func_name, check_task, check_items, check, **kwargs).run()
+        return res, check_result
+
+    @property
+    def role_name(self):
+        return self.role.name
+
+    def role_grant(self, object: str, object_name: str, privilege: str, db_name: str = "", check_task=None, check_items=None, **kwargs):
+        func_name = sys._getframe().f_code.co_name
+        res, check = api_request([self.role.grant, object, object_name, privilege, db_name], **kwargs)
+        check_result = ResponseChecker(res, func_name, check_task, check_items, check, **kwargs).run()
+        return res, check_result
+
+    def role_revoke(self, object: str, object_name: str, privilege: str, db_name: str = "", check_task=None, check_items=None, **kwargs):
+        func_name = sys._getframe().f_code.co_name
+        res, check = api_request([self.role.revoke, object, object_name, privilege, db_name], **kwargs)
+        check_result = ResponseChecker(res, func_name, check_task, check_items, check, **kwargs).run()
+        return res, check_result
+
+    def role_grant_v2(self, privilege: str, collection_name: str, db_name: str = None, check_task=None, check_items=None, **kwargs):
+        func_name = sys._getframe().f_code.co_name
+        res, check = api_request([self.role.grant_v2, privilege, collection_name, db_name], **kwargs)
+        check_result = ResponseChecker(res, func_name, check_task, check_items, check, **kwargs).run()
+        return res, check_result
+
+    def role_revoke_v2(self, privilege: str, collection_name: str, db_name: str = None, check_task=None, check_items=None, **kwargs):
+        func_name = sys._getframe().f_code.co_name
+        res, check = api_request([self.role.revoke_v2, privilege, collection_name, db_name], **kwargs)
+        check_result = ResponseChecker(res, func_name, check_task, check_items, check, **kwargs).run()
+        return res, check_result
+
+    def role_list_grant(self, object: str, object_name: str, db_name: str = "", check_task=None, check_items=None, **kwargs):
+        func_name = sys._getframe().f_code.co_name
+        res, check = api_request([self.role.list_grant, object, object_name, db_name], **kwargs)
+        check_result = ResponseChecker(res, func_name, check_task, check_items, check, **kwargs).run()
+        return res, check_result
+
+    def role_list_grants(self, db_name: str = "", check_task=None, check_items=None, **kwargs):
+        func_name = sys._getframe().f_code.co_name
+        res, check = api_request([self.role.list_grants, db_name], **kwargs)
+        check_result = ResponseChecker(res, func_name, check_task, check_items, check, **kwargs).run()
+        return res, check_result
+
+    def create_resource_group(self, name, using="default", timeout=None, check_task=None, check_items=None, **kwargs):
+        func_name = sys._getframe().f_code.co_name
+        res, check = api_request([self.ut.create_resource_group, name, using, timeout], **kwargs)
+        check_result = ResponseChecker(res, func_name, check_task, check_items, check, **kwargs).run()
+        return res, check_result
+
+    def drop_resource_group(self, name, using="default", timeout=None, check_task=None, check_items=None, **kwargs):
+        func_name = sys._getframe().f_code.co_name
+        res, check = api_request([self.ut.drop_resource_group, name, using, timeout], **kwargs)
+        check_result = ResponseChecker(res, func_name, check_task, check_items, check, **kwargs).run()
+        return res, check_result
+
+    def list_resource_groups(self, using="default", timeout=None, check_task=None, check_items=None, **kwargs):
+        func_name = sys._getframe().f_code.co_name
+        res, check = api_request([self.ut.list_resource_groups, using, timeout], **kwargs)
+        check_result = ResponseChecker(res, func_name, check_task, check_items, check, **kwargs).run()
+        return res, check_result
+
+    def describe_resource_group(self, name, using="default", timeout=None, check_task=None, check_items=None, **kwargs):
+        func_name = sys._getframe().f_code.co_name
+        res, check = api_request([self.ut.describe_resource_group, name, using, timeout], **kwargs)
+        check_result = ResponseChecker(res, func_name, check_task, check_items, check, **kwargs).run()
+        return res, check_result
+
+    def update_resource_group(self, config, using="default", timeout=None, check_task=None, check_items=None, **kwargs):
+        func_name = sys._getframe().f_code.co_name
+        res, check = api_request([self.ut.update_resource_groups, config, using, timeout], **kwargs)
+        check_result = ResponseChecker(res, func_name, check_task, check_items, check, **kwargs).run()
+        return res, check_result
+
+    def transfer_node(self, source, target, num_node, using="default", timeout=None, check_task=None, check_items=None,
+                      **kwargs):
+        func_name = sys._getframe().f_code.co_name
+        res, check = api_request([self.ut.transfer_node, source, target, num_node, using, timeout], **kwargs)
+        check_result = ResponseChecker(res, func_name, check_task, check_items, check, **kwargs).run()
+        return res, check_result
+
+    def transfer_replica(self, source, target, collection_name, num_replica, using="default", timeout=None,
+                         check_task=None, check_items=None, **kwargs):
+        func_name = sys._getframe().f_code.co_name
+        res, check = api_request(
+            [self.ut.transfer_replica, source, target, collection_name, num_replica, using, timeout], **kwargs)
+        check_result = ResponseChecker(res, func_name, check_task, check_items, check, **kwargs).run()
+        return res, check_result
+
+    def rename_collection(self, old_collection_name, new_collection_name, new_db_name="", timeout=None,
+                          check_task=None, check_items=None, **kwargs):
+        func_name = sys._getframe().f_code.co_name
+        res, check = api_request([self.ut.rename_collection, old_collection_name, new_collection_name, new_db_name,
+                                  timeout], **kwargs)
+        check_result = ResponseChecker(res, func_name, check_task, check_items, check,
+                                       old_collection_name=old_collection_name, new_collection_name=new_collection_name,
+                                       new_db_name=new_db_name, timeout=timeout, **kwargs).run()
+        return res, check_result
+
+    def flush_all(self, using="default", timeout=None, check_task=None, check_items=None, **kwargs):
+        func_name = sys._getframe().f_code.co_name
+        res, check = api_request([self.ut.flush_all, using, timeout], **kwargs)
+        check_result = ResponseChecker(res, func_name, check_task, check_items, check,
+                                       using=using, timeout=timeout, **kwargs).run()
+        return res, check_result
+
+    def get_server_type(self, using="default", check_task=None, check_items=None, **kwargs):
+        func_name = sys._getframe().f_code.co_name
+        res, check = api_request([self.ut.get_server_type, using], **kwargs)
+        check_result = ResponseChecker(res, func_name, check_task, check_items, check,
+                                       using=using, **kwargs).run()
+        return res, check_result
+
+    def list_indexes(self, collection_name, using="default", timeout=None, check_task=None, check_items=None, **kwargs):
+        func_name = sys._getframe().f_code.co_name
+        res, check = api_request([self.ut.list_indexes, collection_name, using, timeout], **kwargs)
+        check_result = ResponseChecker(res, func_name, check_task, check_items, check,
+                                       collection_name=collection_name, using=using, timeout=timeout, **kwargs).run()
+        return res, check_result
+
+    def create_privilege_group(self, privilege_group: str, check_task=None, check_items=None, **kwargs):
+        func_name = sys._getframe().f_code.co_name
+        res, check = api_request([self.role.create_privilege_group, privilege_group], **kwargs)
+        check_result = ResponseChecker(res, func_name, check_task, check_items, check, **kwargs).run()
+        return res, check_result
+
+    def drop_privilege_group(self, privilege_group: str, check_task=None, check_items=None, **kwargs):
+        func_name = sys._getframe().f_code.co_name
+        res, check = api_request([self.role.drop_privilege_group, privilege_group], **kwargs)
+        check_result = ResponseChecker(res, func_name, check_task, check_items, check, **kwargs).run()
+        return res, check_result
+
+    def list_privilege_groups(self, check_task=None, check_items=None, **kwargs):
+        func_name = sys._getframe().f_code.co_name
+        res, check = api_request([self.role.list_privilege_groups], **kwargs)
+        check_result = ResponseChecker(res, func_name, check_task, check_items, check, **kwargs).run()
+        return res, check_result
+
+    def add_privileges_to_group(self, privilege_group: str, privileges: list, check_task=None, check_items=None, **kwargs):
+        func_name = sys._getframe().f_code.co_name
+        res, check = api_request([self.role.add_privileges_to_group, privilege_group, privileges], **kwargs)
+        check_result = ResponseChecker(res, func_name, check_task, check_items, check, **kwargs).run()
+        return res, check_result
+
+    def remove_privileges_from_group(self, privilege_group: str, privileges: list, check_task=None, check_items=None, **kwargs):
+        func_name = sys._getframe().f_code.co_name
+        res, check = api_request([self.role.remove_privileges_from_group, privilege_group, privileges], **kwargs)
+        check_result = ResponseChecker(res, func_name, check_task, check_items, check, **kwargs).run()
+        return res, check_result

tests/python_client/bulk_insert/conftest.py

@@ -0,0 +1,35 @@
+import pytest
+
+
+def pytest_addoption(parser):
+    parser.addoption("--file_type", action="store", default="json", help="filetype")
+    parser.addoption("--create_index", action="store",  default="create_index", help="whether creating index")
+    parser.addoption("--nb", action="store", default=50000, help="nb")
+    parser.addoption("--dim", action="store",  default=768, help="dim")
+    parser.addoption("--varchar_len", action="store", default=2000, help="varchar_len")
+    parser.addoption("--with_varchar_field", action="store", default="true", help="with varchar field or not")
+
+@pytest.fixture
+def file_type(request):
+    return request.config.getoption("--file_type")
+
+
+@pytest.fixture
+def create_index(request):
+    return request.config.getoption("--create_index")
+
+@pytest.fixture
+def nb(request):
+    return request.config.getoption("--nb")
+
+@pytest.fixture
+def dim(request):
+    return request.config.getoption("--dim")
+
+@pytest.fixture
+def varchar_len(request):
+    return request.config.getoption("--varchar_len")
+
+@pytest.fixture
+def with_varchar_field(request):
+    return request.config.getoption("--with_varchar_field")

tests/python_client/bulk_insert/test_bulk_insert_bench.py

@@ -0,0 +1,407 @@
+import logging
+import time
+import pytest
+from pymilvus import DataType
+import numpy as np
+from pathlib import Path
+from base.client_base import TestcaseBase
+from common import common_func as cf
+from common import common_type as ct
+from common.milvus_sys import MilvusSys
+from common.common_type import CaseLabel, CheckTasks
+from utils.util_log import test_log as log
+from common.bulk_insert_data import (
+    prepare_bulk_insert_json_files,
+    prepare_bulk_insert_new_json_files,
+    prepare_bulk_insert_numpy_files,
+    prepare_bulk_insert_parquet_files,
+    prepare_bulk_insert_csv_files,
+    DataField as df,
+)
+import json
+import requests
+import time
+import uuid
+from utils.util_log import test_log as logger
+from minio import Minio
+from minio.error import S3Error
+
+
+def logger_request_response(response, url, tt, headers, data, str_data, str_response, method):
+    if len(data) > 2000:
+        data = data[:1000] + "..." + data[-1000:]
+    try:
+        if response.status_code == 200:
+            if ('code' in response.json() and response.json()["code"] == 200) or (
+                    'Code' in response.json() and response.json()["Code"] == 0):
+                logger.debug(
+                    f"\nmethod: {method}, \nurl: {url}, \ncost time: {tt}, \nheader: {headers}, \npayload: {str_data}, \nresponse: {str_response}")
+            else:
+                logger.debug(
+                    f"\nmethod: {method}, \nurl: {url}, \ncost time: {tt}, \nheader: {headers}, \npayload: {data}, \nresponse: {response.text}")
+        else:
+            logger.debug(
+                f"method: \nmethod: {method}, \nurl: {url}, \ncost time: {tt}, \nheader: {headers}, \npayload: {data}, \nresponse: {response.text}")
+    except Exception as e:
+        logger.debug(
+            f"method: \nmethod: {method}, \nurl: {url}, \ncost time: {tt}, \nheader: {headers}, \npayload: {data}, \nresponse: {response.text}, \nerror: {e}")
+
+
+class Requests:
+    def __init__(self, url=None, api_key=None):
+        self.url = url
+        self.api_key = api_key
+        self.headers = {
+            'Content-Type': 'application/json',
+            'Authorization': f'Bearer {self.api_key}',
+            'RequestId': str(uuid.uuid1())
+        }
+
+    def update_headers(self):
+        headers = {
+            'Content-Type': 'application/json',
+            'Authorization': f'Bearer {self.api_key}',
+            'RequestId': str(uuid.uuid1())
+        }
+        return headers
+
+    def post(self, url, headers=None, data=None, params=None):
+        headers = headers if headers is not None else self.update_headers()
+        data = json.dumps(data)
+        str_data = data[:200] + '...' + data[-200:] if len(data) > 400 else data
+        t0 = time.time()
+        response = requests.post(url, headers=headers, data=data, params=params)
+        tt = time.time() - t0
+        str_response = response.text[:200] + '...' + response.text[-200:] if len(response.text) > 400 else response.text
+        logger_request_response(response, url, tt, headers, data, str_data, str_response, "post")
+        return response
+
+    def get(self, url, headers=None, params=None, data=None):
+        headers = headers if headers is not None else self.update_headers()
+        data = json.dumps(data)
+        str_data = data[:200] + '...' + data[-200:] if len(data) > 400 else data
+        t0 = time.time()
+        if data is None or data == "null":
+            response = requests.get(url, headers=headers, params=params)
+        else:
+            response = requests.get(url, headers=headers, params=params, data=data)
+        tt = time.time() - t0
+        str_response = response.text[:200] + '...' + response.text[-200:] if len(response.text) > 400 else response.text
+        logger_request_response(response, url, tt, headers, data, str_data, str_response, "get")
+        return response
+
+    def put(self, url, headers=None, data=None):
+        headers = headers if headers is not None else self.update_headers()
+        data = json.dumps(data)
+        str_data = data[:200] + '...' + data[-200:] if len(data) > 400 else data
+        t0 = time.time()
+        response = requests.put(url, headers=headers, data=data)
+        tt = time.time() - t0
+        str_response = response.text[:200] + '...' + response.text[-200:] if len(response.text) > 400 else response.text
+        logger_request_response(response, url, tt, headers, data, str_data, str_response, "put")
+        return response
+
+    def delete(self, url, headers=None, data=None):
+        headers = headers if headers is not None else self.update_headers()
+        data = json.dumps(data)
+        str_data = data[:200] + '...' + data[-200:] if len(data) > 400 else data
+        t0 = time.time()
+        response = requests.delete(url, headers=headers, data=data)
+        tt = time.time() - t0
+        str_response = response.text[:200] + '...' + response.text[-200:] if len(response.text) > 400 else response.text
+        logger_request_response(response, url, tt, headers, data, str_data, str_response, "delete")
+        return response
+
+
+class ImportJobClient(Requests):
+
+    def __init__(self, endpoint, token):
+        super().__init__(url=endpoint, api_key=token)
+        self.endpoint = endpoint
+        self.api_key = token
+        self.db_name = None
+        self.headers = self.update_headers()
+
+    def update_headers(self):
+        headers = {
+            'Content-Type': 'application/json',
+            'Authorization': f'Bearer {self.api_key}',
+            'RequestId': str(uuid.uuid1())
+        }
+        return headers
+
+    def list_import_jobs(self, payload, db_name="default"):
+        payload["dbName"] = db_name
+        data = payload
+        url = f'{self.endpoint}/v2/vectordb/jobs/import/list'
+        response = self.post(url, headers=self.update_headers(), data=data)
+        res = response.json()
+        return res
+
+    def create_import_jobs(self, payload):
+        url = f'{self.endpoint}/v2/vectordb/jobs/import/create'
+        response = self.post(url, headers=self.update_headers(), data=payload)
+        res = response.json()
+        return res
+
+    def get_import_job_progress(self, task_id):
+        payload = {
+            "jobId": task_id
+        }
+        url = f'{self.endpoint}/v2/vectordb/jobs/import/get_progress'
+        response = self.post(url, headers=self.update_headers(), data=payload)
+        res = response.json()
+        return res
+
+    def wait_import_job_completed(self, task_id_list, timeout=1800):
+        success = False
+        success_states = {}
+        t0 = time.time()
+        while time.time() - t0 < timeout:
+            for task_id in task_id_list:
+                res = self.get_import_job_progress(task_id)
+                if res['data']['state'] == "Completed":
+                    success_states[task_id] = True
+                else:
+                    success_states[task_id] = False
+                time.sleep(5)
+            # all task success then break
+            if all(success_states.values()):
+                success = True
+                break
+        states = []
+        for task_id in task_id_list:
+            res = self.get_import_job_progress(task_id)
+            states.append({
+                "task_id": task_id,
+                "state": res['data']
+            })
+        return success, states
+
+
+default_vec_only_fields = [df.vec_field]
+default_multi_fields = [
+    df.vec_field,
+    df.int_field,
+    df.string_field,
+    df.bool_field,
+    df.float_field,
+    df.array_int_field
+]
+default_vec_n_int_fields = [df.vec_field, df.int_field, df.array_int_field]
+
+
+# milvus_ns = "chaos-testing"
+base_dir = "/tmp/bulk_insert_data"
+
+
+def entity_suffix(entities):
+    if entities // 1000000 > 0:
+        suffix = f"{entities // 1000000}m"
+    elif entities // 1000 > 0:
+        suffix = f"{entities // 1000}k"
+    else:
+        suffix = f"{entities}"
+    return suffix
+
+
+class TestcaseBaseBulkInsert(TestcaseBase):
+    import_job_client = None
+    @pytest.fixture(scope="function", autouse=True)
+    def init_minio_client(self, minio_host):
+        Path("/tmp/bulk_insert_data").mkdir(parents=True, exist_ok=True)
+        self._connect()
+        self.milvus_sys = MilvusSys(alias='default')
+        ms = MilvusSys()
+        minio_port = "9000"
+        self.minio_endpoint = f"{minio_host}:{minio_port}"
+        self.bucket_name = ms.data_nodes[0]["infos"]["system_configurations"][
+            "minio_bucket_name"
+        ]
+
+    @pytest.fixture(scope="function", autouse=True)
+    def init_import_client(self, host, port, user, password):
+        self.import_job_client = ImportJobClient(f"http://{host}:{port}", f"{user}:{password}")
+
+
+class TestBulkInsertPerf(TestcaseBaseBulkInsert):
+
+    @pytest.mark.tags(CaseLabel.L3)
+    @pytest.mark.parametrize("auto_id", [True])
+    @pytest.mark.parametrize("dim", [128])  # 128
+    @pytest.mark.parametrize("file_size", [1, 10, 15])  # file size in GB
+    @pytest.mark.parametrize("file_nums", [1])
+    @pytest.mark.parametrize("array_len", [100])
+    @pytest.mark.parametrize("enable_dynamic_field", [False])
+    def test_bulk_insert_all_field_with_parquet(self, auto_id, dim, file_size, file_nums, array_len, enable_dynamic_field):
+        """
+        collection schema 1: [pk, int64, float64, string float_vector]
+        data file: vectors.parquet and uid.parquet,
+        Steps:
+        1. create collection
+        2. import data
+        3. verify
+        """
+        fields = [
+            cf.gen_int64_field(name=df.pk_field, is_primary=True, auto_id=auto_id),
+            cf.gen_int64_field(name=df.int_field),
+            cf.gen_float_field(name=df.float_field),
+            cf.gen_double_field(name=df.double_field),
+            cf.gen_json_field(name=df.json_field),
+            cf.gen_array_field(name=df.array_int_field, element_type=DataType.INT64),
+            cf.gen_array_field(name=df.array_float_field, element_type=DataType.FLOAT),
+            cf.gen_array_field(name=df.array_string_field, element_type=DataType.VARCHAR, max_length=200),
+            cf.gen_array_field(name=df.array_bool_field, element_type=DataType.BOOL),
+            cf.gen_float_vec_field(name=df.vec_field, dim=dim),
+        ]
+        data_fields = [f.name for f in fields if not f.to_dict().get("auto_id", False)]
+        files = prepare_bulk_insert_parquet_files(
+            minio_endpoint=self.minio_endpoint,
+            bucket_name=self.bucket_name,
+            rows=3000,
+            dim=dim,
+            data_fields=data_fields,
+            file_size=file_size,
+            row_group_size=None,
+            file_nums=file_nums,
+            array_length=array_len,
+            enable_dynamic_field=enable_dynamic_field,
+            force=True,
+        )
+        self._connect()
+        c_name = cf.gen_unique_str("bulk_insert")
+        schema = cf.gen_collection_schema(fields=fields, auto_id=auto_id, enable_dynamic_field=enable_dynamic_field)
+        self.collection_wrap.init_collection(c_name, schema=schema)
+        payload = {
+            "collectionName": c_name,
+            "files": [files],
+        }
+
+        # import data
+        payload = {
+            "collectionName": c_name,
+            "files": [files],
+        }
+        t0 = time.time()
+        rsp = self.import_job_client.create_import_jobs(payload)
+        job_id_list = [rsp["data"]["jobId"]]
+        logging.info(f"bulk insert job ids:{job_id_list}")
+        success, states = self.import_job_client.wait_import_job_completed(job_id_list, timeout=1800)
+        tt = time.time() - t0
+        log.info(f"bulk insert state:{success} in {tt} with states:{states}")
+        assert success
+
+    @pytest.mark.tags(CaseLabel.L3)
+    @pytest.mark.parametrize("auto_id", [True])
+    @pytest.mark.parametrize("dim", [128])  # 128
+    @pytest.mark.parametrize("file_size", [1, 10, 15])  # file size in GB
+    @pytest.mark.parametrize("file_nums", [1])
+    @pytest.mark.parametrize("array_len", [100])
+    @pytest.mark.parametrize("enable_dynamic_field", [False])
+    def test_bulk_insert_all_field_with_json(self, auto_id, dim, file_size, file_nums, array_len, enable_dynamic_field):
+        """
+        collection schema 1: [pk, int64, float64, string float_vector]
+        data file: vectors.parquet and uid.parquet,
+        Steps:
+        1. create collection
+        2. import data
+        3. verify
+        """
+        fields = [
+            cf.gen_int64_field(name=df.pk_field, is_primary=True, auto_id=auto_id),
+            cf.gen_int64_field(name=df.int_field),
+            cf.gen_float_field(name=df.float_field),
+            cf.gen_double_field(name=df.double_field),
+            cf.gen_json_field(name=df.json_field),
+            cf.gen_array_field(name=df.array_int_field, element_type=DataType.INT64),
+            cf.gen_array_field(name=df.array_float_field, element_type=DataType.FLOAT),
+            cf.gen_array_field(name=df.array_string_field, element_type=DataType.VARCHAR, max_length=200),
+            cf.gen_array_field(name=df.array_bool_field, element_type=DataType.BOOL),
+            cf.gen_float_vec_field(name=df.vec_field, dim=dim),
+        ]
+        data_fields = [f.name for f in fields if not f.to_dict().get("auto_id", False)]
+        files = prepare_bulk_insert_new_json_files(
+            minio_endpoint=self.minio_endpoint,
+            bucket_name=self.bucket_name,
+            rows=3000,
+            dim=dim,
+            data_fields=data_fields,
+            file_size=file_size,
+            file_nums=file_nums,
+            array_length=array_len,
+            enable_dynamic_field=enable_dynamic_field,
+            force=True,
+        )
+        self._connect()
+        c_name = cf.gen_unique_str("bulk_insert")
+        schema = cf.gen_collection_schema(fields=fields, auto_id=auto_id, enable_dynamic_field=enable_dynamic_field)
+        self.collection_wrap.init_collection(c_name, schema=schema)
+
+        # import data
+        payload = {
+            "collectionName": c_name,
+            "files": [files],
+        }
+        t0 = time.time()
+        rsp = self.import_job_client.create_import_jobs(payload)
+        job_id_list = [rsp["data"]["jobId"]]
+        logging.info(f"bulk insert job ids:{job_id_list}")
+        success, states = self.import_job_client.wait_import_job_completed(job_id_list, timeout=1800)
+        tt = time.time() - t0
+        log.info(f"bulk insert state:{success} in {tt} with states:{states}")
+        assert success
+
+
+    @pytest.mark.tags(CaseLabel.L3)
+    @pytest.mark.parametrize("auto_id", [True])
+    @pytest.mark.parametrize("dim", [128])  # 128
+    @pytest.mark.parametrize("file_size", [1, 10, 15])  # file size in GB
+    @pytest.mark.parametrize("file_nums", [1])
+    @pytest.mark.parametrize("enable_dynamic_field", [False])
+    def test_bulk_insert_all_field_with_numpy(self, auto_id, dim, file_size, file_nums, enable_dynamic_field):
+        """
+        collection schema 1: [pk, int64, float64, string float_vector]
+        data file: vectors.parquet and uid.parquet,
+        Steps:
+        1. create collection
+        2. import data
+        3. verify
+        """
+        fields = [
+            cf.gen_int64_field(name=df.pk_field, is_primary=True, auto_id=auto_id),
+            cf.gen_int64_field(name=df.int_field),
+            cf.gen_float_field(name=df.float_field),
+            cf.gen_double_field(name=df.double_field),
+            cf.gen_json_field(name=df.json_field),
+            cf.gen_float_vec_field(name=df.vec_field, dim=dim),
+        ]
+        data_fields = [f.name for f in fields if not f.to_dict().get("auto_id", False)]
+        files = prepare_bulk_insert_numpy_files(
+            minio_endpoint=self.minio_endpoint,
+            bucket_name=self.bucket_name,
+            rows=3000,
+            dim=dim,
+            data_fields=data_fields,
+            file_size=file_size,
+            file_nums=file_nums,
+            enable_dynamic_field=enable_dynamic_field,
+            force=True,
+        )
+        self._connect()
+        c_name = cf.gen_unique_str("bulk_insert")
+        schema = cf.gen_collection_schema(fields=fields, auto_id=auto_id, enable_dynamic_field=enable_dynamic_field)
+        self.collection_wrap.init_collection(c_name, schema=schema)
+
+        # import data
+        payload = {
+            "collectionName": c_name,
+            "files": [files],
+        }
+        t0 = time.time()
+        rsp = self.import_job_client.create_import_jobs(payload)
+        job_id_list = [rsp["data"]["jobId"]]
+        logging.info(f"bulk insert job ids:{job_id_list}")
+        success, states = self.import_job_client.wait_import_job_completed(job_id_list, timeout=1800)
+        tt = time.time() - t0
+        log.info(f"bulk insert state:{success} in {tt} with states:{states}")
+        assert success

tests/python_client/bulk_insert/test_bulk_insert_perf.py

@@ -0,0 +1,154 @@
+import pytest
+import os
+import time
+import threading
+from pathlib import Path
+from time import sleep
+from minio import Minio
+from pymilvus import connections
+from chaos.checker import (BulkInsertChecker, Op)
+from common.milvus_sys import MilvusSys
+from utils.util_log import test_log as log
+from utils.util_k8s import get_milvus_deploy_tool, get_pod_ip_name_pairs, get_milvus_instance_name
+from chaos import chaos_commons as cc
+from common.common_type import CaseLabel
+from common import common_func as cf
+from chaos import constants
+from delayed_assert import expect, assert_expectations
+
+
+def assert_statistic(checkers, expectations={}):
+    for k in checkers.keys():
+        # expect succ if no expectations
+        succ_rate = checkers[k].succ_rate()
+        total = checkers[k].total()
+        average_time = checkers[k].average_time
+        if expectations.get(k, '') == constants.FAIL:
+            log.info(
+                f"Expect Fail: {str(k)} succ rate {succ_rate}, total: {total}, average time: {average_time:.4f}")
+            expect(succ_rate < 0.49 or total < 2,
+                   f"Expect Fail: {str(k)} succ rate {succ_rate}, total: {total}, average time: {average_time:.4f}")
+        else:
+            log.info(
+                f"Expect Succ: {str(k)} succ rate {succ_rate}, total: {total}, average time: {average_time:.4f}")
+            expect(succ_rate > 0.90 and total > 2,
+                   f"Expect Succ: {str(k)} succ rate {succ_rate}, total: {total}, average time: {average_time:.4f}")
+
+
+def get_querynode_info(release_name):
+    querynode_id_pod_pair = {}
+    querynode_ip_pod_pair = get_pod_ip_name_pairs(
+        "chaos-testing", f"app.kubernetes.io/instance={release_name}, component=querynode")
+    ms = MilvusSys()
+    for node in ms.query_nodes:
+        ip = node["infos"]['hardware_infos']["ip"].split(":")[0]
+        querynode_id_pod_pair[node["identifier"]] = querynode_ip_pod_pair[ip]
+    return querynode_id_pod_pair
+
+
+class TestChaosBase:
+    expect_create = constants.SUCC
+    expect_insert = constants.SUCC
+    expect_flush = constants.SUCC
+    expect_index = constants.SUCC
+    expect_search = constants.SUCC
+    expect_query = constants.SUCC
+    host = '127.0.0.1'
+    port = 19530
+    _chaos_config = None
+    health_checkers = {}
+
+
+class TestChaos(TestChaosBase):
+
+    def teardown_method(self):
+        sleep(10)
+        log.info(f'Alive threads: {threading.enumerate()}')
+
+    @pytest.fixture(scope="function", autouse=True)
+    def connection(self, host, port, milvus_ns):
+        connections.add_connection(default={"host": host, "port": port})
+        connections.connect(alias='default')
+
+        if connections.has_connection("default") is False:
+            raise Exception("no connections")
+        instance_name = get_milvus_instance_name(constants.CHAOS_NAMESPACE, host)
+        self.host = host
+        self.port = port
+        self.instance_name = instance_name
+        self.milvus_sys = MilvusSys(alias='default')
+        self.milvus_ns = milvus_ns
+        self.release_name = get_milvus_instance_name(self.milvus_ns, milvus_sys=self.milvus_sys)
+        self.deploy_by = get_milvus_deploy_tool(self.milvus_ns, self.milvus_sys)
+
+    def init_health_checkers(self, collection_name=None, dim=2048):
+        log.info("init health checkers")
+        c_name = collection_name if collection_name else cf.gen_unique_str("Checker_")
+        checkers = {
+            Op.bulk_insert: BulkInsertChecker(collection_name=c_name, use_one_collection=False, dim=dim,),
+        }
+        self.health_checkers = checkers
+
+    def prepare_bulk_insert(self, nb=3000, file_type="json", dim=768, varchar_len=2000, with_varchar_field=True):
+        if Op.bulk_insert not in self.health_checkers:
+            log.info("bulk_insert checker is not in  health checkers, skip prepare bulk load")
+            return
+        log.info("bulk_insert checker is in health checkers, prepare data firstly")
+        deploy_tool = get_milvus_deploy_tool(self.milvus_ns, self.milvus_sys)
+        if deploy_tool == "helm":
+            release_name = self.instance_name
+        else:
+            release_name = self.instance_name + "-minio"
+        minio_ip_pod_pair = get_pod_ip_name_pairs("chaos-testing", f"release={release_name}, app=minio")
+        ms = MilvusSys()
+        minio_ip = list(minio_ip_pod_pair.keys())[0]
+        minio_port = "9000"
+        minio_endpoint = f"{minio_ip}:{minio_port}"
+        bucket_name = ms.data_nodes[0]["infos"]["system_configurations"]["minio_bucket_name"]
+        schema = cf.gen_bulk_insert_collection_schema(dim=dim, with_varchar_field=with_varchar_field)
+        data = cf.gen_default_list_data_for_bulk_insert(nb=nb, varchar_len=varchar_len,
+                                                        with_varchar_field=with_varchar_field)
+        data_dir = "/tmp/bulk_insert_data"
+        Path(data_dir).mkdir(parents=True, exist_ok=True)
+        files = []
+        if file_type == "json":
+            files = cf.gen_json_files_for_bulk_insert(data, schema, data_dir, nb=nb, dim=dim)
+        if file_type == "npy":
+            files = cf.gen_npy_files_for_bulk_insert(data, schema, data_dir, nb=nb, dim=dim)
+        log.info("upload file to minio")
+        client = Minio(minio_endpoint, access_key="minioadmin", secret_key="minioadmin", secure=False)
+        for file_name in files:
+            file_size = os.path.getsize(os.path.join(data_dir, file_name)) / 1024 / 1024
+            t0 = time.time()
+            client.fput_object(bucket_name, file_name, os.path.join(data_dir, file_name))
+            log.info(f"upload file {file_name} to minio, size: {file_size:.2f} MB, cost {time.time() - t0:.2f} s")
+        self.health_checkers[Op.bulk_insert].update(schema=schema, files=files)
+        log.info("prepare data for bulk load done")
+
+    @pytest.mark.tags(CaseLabel.L3)
+    def test_bulk_insert_perf(self, file_type, nb, dim, varchar_len, with_varchar_field):
+        # start the monitor threads to check the milvus ops
+        log.info("*********************Test Start**********************")
+        log.info(connections.get_connection_addr('default'))
+        log.info(f"file_type: {file_type}, nb: {nb}, dim: {dim}, varchar_len: {varchar_len}, with_varchar_field: {with_varchar_field}")
+        self.init_health_checkers(dim=int(dim))
+        nb = int(nb)
+        if str(with_varchar_field) in ["true", "True"]:
+            with_varchar_field = True
+        else:
+            with_varchar_field = False
+        varchar_len = int(varchar_len)
+
+        self.prepare_bulk_insert(file_type=file_type, nb=nb, dim=int(dim), varchar_len=varchar_len, with_varchar_field=with_varchar_field)
+        cc.start_monitor_threads(self.health_checkers)
+        # wait 600s
+        while self.health_checkers[Op.bulk_insert].total() <= 10:
+            sleep(constants.WAIT_PER_OP)
+        assert_statistic(self.health_checkers)
+
+        assert_expectations()
+        for k, checker in self.health_checkers.items():
+            checker.check_result()
+            checker.terminate()
+
+        log.info("*********************Test Completed**********************")

tests/python_client/bulk_insert/test_bulk_insert_perf_with_cohere_dataset.py

@@ -0,0 +1,106 @@
+import pytest
+import threading
+from time import sleep
+from pymilvus import connections, DataType, FieldSchema, CollectionSchema
+from chaos.checker import (BulkInsertChecker, Op)
+from utils.util_log import test_log as log
+from chaos import chaos_commons as cc
+from common.common_type import CaseLabel
+from common import common_func as cf
+from chaos import constants
+from delayed_assert import expect, assert_expectations
+
+
+def assert_statistic(checkers, expectations={}):
+    for k in checkers.keys():
+        # expect succ if no expectations
+        succ_rate = checkers[k].succ_rate()
+        total = checkers[k].total()
+        average_time = checkers[k].average_time
+        if expectations.get(k, '') == constants.FAIL:
+            log.info(
+                f"Expect Fail: {str(k)} succ rate {succ_rate}, total: {total}, average time: {average_time:.4f}")
+            expect(succ_rate < 0.49 or total < 2,
+                   f"Expect Fail: {str(k)} succ rate {succ_rate}, total: {total}, average time: {average_time:.4f}")
+        else:
+            log.info(
+                f"Expect Succ: {str(k)} succ rate {succ_rate}, total: {total}, average time: {average_time:.4f}")
+            expect(succ_rate > 0.90 and total > 2,
+                   f"Expect Succ: {str(k)} succ rate {succ_rate}, total: {total}, average time: {average_time:.4f}")
+
+
+class TestBulkInsertBase:
+    expect_create = constants.SUCC
+    expect_insert = constants.SUCC
+    expect_flush = constants.SUCC
+    expect_index = constants.SUCC
+    expect_search = constants.SUCC
+    expect_query = constants.SUCC
+    host = '127.0.0.1'
+    port = 19530
+    _chaos_config = None
+    health_checkers = {}
+
+
+class TestBUlkInsertPerf(TestBulkInsertBase):
+
+    def teardown_method(self):
+        sleep(10)
+        log.info(f'Alive threads: {threading.enumerate()}')
+
+    @pytest.fixture(scope="function", autouse=True)
+    def connection(self, host, port, milvus_ns):
+        connections.add_connection(default={"host": host, "port": port})
+        connections.connect(alias='default')
+
+        if connections.has_connection("default") is False:
+            raise Exception("no connections")
+
+    def init_health_checkers(self, collection_name=None, file_type="npy"):
+        log.info("init health checkers")
+        c_name = collection_name if collection_name else cf.gen_unique_str("BulkInsertChecker")
+        fields = [
+            FieldSchema(name="id", dtype=DataType.INT64, is_primary=True),
+            FieldSchema(name="title", dtype=DataType.VARCHAR, max_length=65535),
+            FieldSchema(name="text", dtype=DataType.VARCHAR, max_length=65535),
+            FieldSchema(name="url", dtype=DataType.VARCHAR, max_length=65535),
+            FieldSchema(name="wiki_id", dtype=DataType.INT64),
+            FieldSchema(name="views", dtype=DataType.DOUBLE),
+            FieldSchema(name="paragraph_id", dtype=DataType.INT64),
+            FieldSchema(name="langs", dtype=DataType.INT64),
+            FieldSchema(name="emb", dtype=DataType.FLOAT_VECTOR, dim=768)
+        ]
+        schema = CollectionSchema(fields=fields, description="test collection")
+        fields_name = ["id", "title", "text", "url", "wiki_id", "views", "paragraph_id", "langs", "emb"]
+        files = []
+        if file_type == "json":
+            files = ["train-00000-of-00252.json"]
+        if file_type == "npy":
+            for field_name in fields_name:
+                files.append(f"{field_name}.npy")
+        if file_type == "parquet":
+            files = ["train-00000-of-00252.parquet"]
+        checkers = {
+            Op.bulk_insert: BulkInsertChecker(collection_name=c_name, use_one_collection=False, schema=schema,
+                                              files=files, insert_data=False)
+        }
+        self.health_checkers = checkers
+
+    @pytest.mark.tags(CaseLabel.L3)
+    def test_bulk_insert_perf(self):
+        # start the monitor threads to check the milvus ops
+        log.info("*********************Test Start**********************")
+        log.info(connections.get_connection_addr('default'))
+        self.init_health_checkers()
+        cc.start_monitor_threads(self.health_checkers)
+        # wait 600s
+        while self.health_checkers[Op.bulk_insert].total() <= 10:
+            sleep(constants.WAIT_PER_OP)
+        assert_statistic(self.health_checkers)
+
+        assert_expectations()
+        for k, checker in self.health_checkers.items():
+            checker.check_result()
+            checker.terminate()
+
+        log.info("*********************Test Completed**********************")

tests/python_client/bulk_insert/test_bulk_insert_task_clean.py

@@ -0,0 +1,248 @@
+import logging
+import time
+import pytest
+from pathlib import Path
+from base.client_base import TestcaseBase
+from common import common_func as cf
+from common import common_type as ct
+from common.milvus_sys import MilvusSys
+from common.common_type import CaseLabel, CheckTasks
+from utils.util_k8s import (
+    get_pod_ip_name_pairs,
+    get_milvus_instance_name,
+    get_milvus_deploy_tool
+)
+from utils.util_log import test_log as log
+from common.bulk_insert_data import (
+    prepare_bulk_insert_json_files,
+    DataField as df,
+    DataErrorType,
+)
+
+
+default_vec_only_fields = [df.vec_field]
+default_multi_fields = [
+    df.vec_field,
+    df.int_field,
+    df.string_field,
+    df.bool_field,
+    df.float_field,
+]
+default_vec_n_int_fields = [df.vec_field, df.int_field]
+
+
+milvus_ns = "chaos-testing"
+base_dir = "/tmp/bulk_insert_data"
+
+
+def entity_suffix(entities):
+    if entities // 1000000 > 0:
+        suffix = f"{entities // 1000000}m"
+    elif entities // 1000 > 0:
+        suffix = f"{entities // 1000}k"
+    else:
+        suffix = f"{entities}"
+    return suffix
+
+
+class TestcaseBaseBulkInsert(TestcaseBase):
+
+    @pytest.fixture(scope="function", autouse=True)
+    def init_minio_client(self, host, milvus_ns):
+        Path("/tmp/bulk_insert_data").mkdir(parents=True, exist_ok=True)
+        self._connect()
+        self.milvus_ns = milvus_ns
+        self.milvus_sys = MilvusSys(alias='default')
+        self.instance_name = get_milvus_instance_name(self.milvus_ns, host)
+        self.deploy_tool = get_milvus_deploy_tool(self.milvus_ns, self.milvus_sys)
+        minio_label = f"release={self.instance_name}, app=minio"
+        if self.deploy_tool == "milvus-operator":
+            minio_label = f"release={self.instance_name}-minio, app=minio"
+        minio_ip_pod_pair = get_pod_ip_name_pairs(
+            self.milvus_ns, minio_label
+        )
+        ms = MilvusSys()
+        minio_ip = list(minio_ip_pod_pair.keys())[0]
+        minio_port = "9000"
+        self.minio_endpoint = f"{minio_ip}:{minio_port}"
+        self.bucket_name = ms.data_nodes[0]["infos"]["system_configurations"][
+            "minio_bucket_name"
+        ]
+
+    # def teardown_method(self, method):
+    #     log.info(("*" * 35) + " teardown " + ("*" * 35))
+    #     log.info("[teardown_method] Start teardown test case %s..." % method.__name__)
+
+
+class TestBulkInsertTaskClean(TestcaseBaseBulkInsert):
+
+    @pytest.mark.tags(CaseLabel.L3)
+    @pytest.mark.parametrize("is_row_based", [True])
+    @pytest.mark.parametrize("auto_id", [True, False])
+    @pytest.mark.parametrize("dim", [8])  # 8, 128
+    @pytest.mark.parametrize("entities", [100])  # 100, 1000
+    def test_success_task_not_cleaned(self, is_row_based, auto_id, dim, entities):
+        """
+        collection: auto_id, customized_id
+        collection schema: [pk, float_vector]
+        Steps:
+        1. create collection
+        2. import data
+        3. verify the data entities equal the import data
+        4. load the collection
+        5. verify search successfully
+        6. verify query successfully
+        7. wait for task clean triggered
+        8. verify the task not cleaned
+        """
+        files = prepare_bulk_insert_json_files(
+            minio_endpoint=self.minio_endpoint,
+            bucket_name=self.bucket_name,
+            is_row_based=is_row_based,
+            rows=entities,
+            dim=dim,
+            auto_id=auto_id,
+            data_fields=default_vec_only_fields,
+            force=True,
+        )
+        self._connect()
+        c_name = cf.gen_unique_str("bulk_insert")
+        fields = [
+            cf.gen_int64_field(name=df.pk_field, is_primary=True),
+            cf.gen_float_vec_field(name=df.vec_field, dim=dim),
+        ]
+        schema = cf.gen_collection_schema(fields=fields, auto_id=auto_id)
+        self.collection_wrap.init_collection(c_name, schema=schema)
+        # import data
+        t0 = time.time()
+        task_id, _ = self.utility_wrap.do_bulk_insert(
+            collection_name=c_name,
+            partition_name=None,
+            files=files,
+        )
+        logging.info(f"bulk insert task ids:{task_id}")
+        success, _ = self.utility_wrap.wait_for_bulk_insert_tasks_completed(
+            task_ids=[task_id], timeout=90
+        )
+        tt = time.time() - t0
+        log.info(f"bulk insert state:{success} in {tt}")
+        assert success
+
+        num_entities = self.collection_wrap.num_entities
+        log.info(f" collection entities: {num_entities}")
+        assert num_entities == entities
+
+        # verify imported data is available for search
+        index_params = ct.default_index
+        self.collection_wrap.create_index(
+            field_name=df.vec_field, index_params=index_params
+        )
+        self.collection_wrap.load()
+        log.info(f"wait for load finished and be ready for search")
+        time.sleep(5)
+        log.info(
+            f"query seg info: {self.utility_wrap.get_query_segment_info(c_name)[0]}"
+        )
+        nq = 2
+        topk = 2
+        search_data = cf.gen_vectors(nq, dim)
+        search_params = ct.default_search_params
+        res, _ = self.collection_wrap.search(
+            search_data,
+            df.vec_field,
+            param=search_params,
+            limit=topk,
+            check_task=CheckTasks.check_search_results,
+            check_items={"nq": nq, "limit": topk},
+        )
+        for hits in res:
+            ids = hits.ids
+            results, _ = self.collection_wrap.query(expr=f"{df.pk_field} in {ids}")
+            assert len(results) == len(ids)
+        log.info("wait for task clean triggered")
+        time.sleep(6*60)  # wait for 6 minutes for task clean triggered
+        num_entities = self.collection_wrap.num_entities
+        log.info(f" collection entities: {num_entities}")
+        assert num_entities == entities
+        res, _ = self.collection_wrap.search(
+            search_data,
+            df.vec_field,
+            param=search_params,
+            limit=topk,
+            check_task=CheckTasks.check_search_results,
+            check_items={"nq": nq, "limit": topk},
+        )
+        for hits in res:
+            ids = hits.ids
+            results, _ = self.collection_wrap.query(expr=f"{df.pk_field} in {ids}")
+            assert len(results) == len(ids)
+
+    @pytest.mark.tags(CaseLabel.L3)
+    @pytest.mark.parametrize("is_row_based", [True])
+    @pytest.mark.parametrize("auto_id", [True, False])
+    @pytest.mark.parametrize("dim", [8])  # 8, 128
+    @pytest.mark.parametrize("entities", [100])  # 100, 1000
+    def test_failed_task_was_cleaned(self, is_row_based, auto_id, dim, entities):
+        """
+        collection: auto_id, customized_id
+        collection schema: [pk, float_vector]
+        Steps:
+        1. create collection
+        2. import data with wrong dimension
+        3. verify the data entities is 0 and task was failed
+        4. wait for task clean triggered
+        5. verify the task was cleaned
+
+        """
+        files = prepare_bulk_insert_json_files(
+            minio_endpoint=self.minio_endpoint,
+            bucket_name=self.bucket_name,
+            is_row_based=is_row_based,
+            rows=entities,
+            dim=dim,
+            auto_id=auto_id,
+            data_fields=default_vec_only_fields,
+            err_type=DataErrorType.one_entity_wrong_dim,
+            wrong_position=entities // 2,
+            force=True,
+        )
+        self._connect()
+        c_name = cf.gen_unique_str("bulk_insert")
+        fields = [
+            cf.gen_int64_field(name=df.pk_field, is_primary=True),
+            cf.gen_float_vec_field(name=df.vec_field, dim=dim),
+        ]
+        schema = cf.gen_collection_schema(fields=fields, auto_id=auto_id)
+        self.collection_wrap.init_collection(c_name, schema=schema)
+        # import data
+        t0 = time.time()
+        task_id, _ = self.utility_wrap.do_bulk_insert(
+            collection_name=c_name,
+            partition_name=None,
+            is_row_based=is_row_based,
+            files=files,
+        )
+        logging.info(f"bulk insert task ids:{task_id}")
+        success, states = self.utility_wrap.wait_for_bulk_insert_tasks_completed(
+            task_ids=[task_id], timeout=90
+        )
+        tt = time.time() - t0
+        log.info(f"bulk insert state:{success} in {tt}")
+        assert not success
+        for state in states.values():
+            assert state.state_name in ["Failed", "Failed and cleaned"]
+
+        num_entities = self.collection_wrap.num_entities
+        log.info(f" collection entities: {num_entities}")
+        assert num_entities == 0
+        log.info("wait for task clean triggered")
+        time.sleep(6*60)  # wait for 6 minutes for task clean triggered
+        num_entities = self.collection_wrap.num_entities
+        log.info(f" collection entities: {num_entities}")
+        assert num_entities == 0
+        success, states = self.utility_wrap.wait_for_bulk_insert_tasks_completed(
+            task_ids=[task_id], timeout=90
+        )
+        assert not success
+        for state in states.values():
+            assert state.state_name in ["Failed and cleaned"]

tests/python_client/bulk_insert/test_bulk_insert_with_requests.py

@@ -0,0 +1,149 @@
+import threading
+
+import pytest
+import os
+import time
+import json
+from time import sleep
+from pathlib import Path
+from minio import Minio
+from pymilvus import connections
+from chaos.checker import (InsertChecker, SearchChecker, QueryChecker, BulkInsertChecker, Op)
+from common.cus_resource_opts import CustomResourceOperations as CusResource
+from common.milvus_sys import MilvusSys
+from utils.util_log import test_log as log
+from utils.util_k8s import wait_pods_ready, get_milvus_deploy_tool, get_pod_ip_name_pairs, get_milvus_instance_name
+from utils.util_common import update_key_value
+from chaos import chaos_commons as cc
+from common.common_type import CaseLabel
+from common import common_func as cf
+from chaos import constants
+from delayed_assert import expect, assert_expectations
+
+
+def assert_statistic(checkers, expectations={}):
+    for k in checkers.keys():
+        # expect succ if no expectations
+        succ_rate = checkers[k].succ_rate()
+        total = checkers[k].total()
+        average_time = checkers[k].average_time
+        if expectations.get(k, '') == constants.FAIL:
+            log.info(
+                f"Expect Fail: {str(k)} succ rate {succ_rate}, total: {total}, average time: {average_time:.4f}")
+            expect(succ_rate < 0.49 or total < 2,
+                   f"Expect Fail: {str(k)} succ rate {succ_rate}, total: {total}, average time: {average_time:.4f}")
+        else:
+            log.info(
+                f"Expect Succ: {str(k)} succ rate {succ_rate}, total: {total}, average time: {average_time:.4f}")
+            expect(succ_rate > 0.90 and total > 2,
+                   f"Expect Succ: {str(k)} succ rate {succ_rate}, total: {total}, average time: {average_time:.4f}")
+
+
+def get_querynode_info(release_name):
+    querynode_id_pod_pair = {}
+    querynode_ip_pod_pair = get_pod_ip_name_pairs(
+        "chaos-testing", f"app.kubernetes.io/instance={release_name}, component=querynode")
+    ms = MilvusSys()
+    for node in ms.query_nodes:
+        ip = node["infos"]['hardware_infos']["ip"].split(":")[0]
+        querynode_id_pod_pair[node["identifier"]] = querynode_ip_pod_pair[ip]
+    return querynode_id_pod_pair
+
+
+class TestChaosBase:
+    expect_create = constants.SUCC
+    expect_insert = constants.SUCC
+    expect_flush = constants.SUCC
+    expect_index = constants.SUCC
+    expect_search = constants.SUCC
+    expect_query = constants.SUCC
+    host = '127.0.0.1'
+    port = 19530
+    _chaos_config = None
+    health_checkers = {}
+
+
+class TestChaos(TestChaosBase):
+
+    @pytest.fixture(scope="function", autouse=True)
+    def connection(self, host, port, milvus_ns):
+        connections.add_connection(default={"host": host, "port": port})
+        connections.connect(alias='default')
+
+        if connections.has_connection("default") is False:
+            raise Exception("no connections")
+        instance_name = get_milvus_instance_name(constants.CHAOS_NAMESPACE, host)
+        self.host = host
+        self.port = port
+        self.instance_name = instance_name
+        self.milvus_sys = MilvusSys(alias='default')
+        self.milvus_ns = milvus_ns
+        self.release_name = get_milvus_instance_name(self.milvus_ns, milvus_sys=self.milvus_sys)
+        self.deploy_by = get_milvus_deploy_tool(self.milvus_ns, self.milvus_sys)
+
+    @pytest.fixture(scope="function", autouse=True)
+    def init_health_checkers(self, collection_name=None):
+        log.info("init health checkers")
+        c_name = collection_name if collection_name else cf.gen_unique_str("Checker_")
+        checkers = {
+            Op.insert: InsertChecker(collection_name=c_name),
+            Op.search: SearchChecker(collection_name=c_name),
+            Op.bulk_insert: BulkInsertChecker(collection_name=c_name, use_one_collection=True),
+            Op.query: QueryChecker(collection_name=c_name)
+        }
+        self.health_checkers = checkers
+
+    @pytest.fixture(scope="function", autouse=True)
+    def prepare_bulk_insert(self, nb=3000):
+        if Op.bulk_insert not in self.health_checkers:
+            log.info("bulk_insert checker is not in  health checkers, skip prepare bulk load")
+            return
+        log.info("bulk_insert checker is in  health checkers, prepare data firstly")
+        deploy_tool = get_milvus_deploy_tool(self.milvus_ns, self.milvus_sys)
+        if deploy_tool == "helm":
+            release_name = self.instance_name
+        else:
+            release_name = self.instance_name + "-minio"
+        minio_ip_pod_pair = get_pod_ip_name_pairs("chaos-testing", f"release={release_name}, app=minio")
+        ms = MilvusSys()
+        minio_ip = list(minio_ip_pod_pair.keys())[0]
+        minio_port = "9000"
+        minio_endpoint = f"{minio_ip}:{minio_port}"
+        bucket_name = ms.data_nodes[0]["infos"]["system_configurations"]["minio_bucket_name"]
+        schema = cf.gen_default_collection_schema()
+        data = cf.gen_default_list_data_for_bulk_insert(nb=nb)
+        fields_name = [field.name for field in schema.fields]
+        entities = []
+        for i in range(nb):
+            entity_value = [field_values[i] for field_values in data]
+            entity = dict(zip(fields_name, entity_value))
+            entities.append(entity)
+        data_dict = {"rows": entities}
+        data_source = "/tmp/ci_logs/bulk_insert_data_source.json"
+        file_name = "bulk_insert_data_source.json"
+        files = ["bulk_insert_data_source.json"]
+        # TODO: npy file type is not supported so far
+        log.info("generate bulk load file")
+        with open(data_source, "w") as f:
+            f.write(json.dumps(data_dict, indent=4))
+        log.info("upload file to minio")
+        client = Minio(minio_endpoint, access_key="minioadmin", secret_key="minioadmin", secure=False)
+        client.fput_object(bucket_name, file_name, data_source)
+        self.health_checkers[Op.bulk_insert].update(schema=schema, files=files)
+        log.info("prepare data for bulk load done")
+
+    @pytest.mark.tags(CaseLabel.L3)
+    def test_bulk_insert(self):
+        # start the monitor threads to check the milvus ops
+        log.info("*********************Test Start**********************")
+        log.info(connections.get_connection_addr('default'))
+        # c_name = cf.gen_unique_str("BulkInsertChecker_")
+        # self.init_health_checkers(collection_name=c_name)
+        cc.start_monitor_threads(self.health_checkers)
+        # wait 120s
+        sleep(constants.WAIT_PER_OP * 12)
+        assert_statistic(self.health_checkers)
+
+        assert_expectations()
+
+        log.info("*********************Test Completed**********************")

tests/python_client/cdc/README.md

@@ -0,0 +1,235 @@
+# CDC Sync Test Cases
+
+## Overview
+
+This test suite validates CDC (Change Data Capture) synchronization between upstream and downstream Milvus clusters. It verifies that operations performed on the upstream cluster are correctly replicated to the downstream cluster through CDC.
+
+**What it tests:**
+- Database, collection, partition, and index operations
+- Data manipulation (insert, delete, upsert, bulk insert)
+- RBAC operations (users, roles, privileges)
+- Collection management (load, release, flush, compact)
+- Alias operations
+
+**Key Features:**
+- Automatic CDC topology setup at test start
+- Query-based verification to ensure data consistency
+- Configurable sync timeout with progress logging
+- Comprehensive test coverage for all CDC-supported operations
+
+## Prerequisites
+
+Before running the tests, ensure you have:
+
+1. **Two running Milvus instances:**
+   - Upstream cluster (source)
+   - Downstream cluster (target)
+
+2. **Python dependencies:**
+   ```bash
+   pip install pymilvus>=2.6.0 pytest numpy
+   ```
+
+3. **Network connectivity:**
+   - Both clusters should be accessible from the test environment
+   - Authentication credentials for both clusters
+
+## Quick Start
+
+The simplest way to run tests with default configuration:
+
+```bash
+cd /path/to/milvus/tests/python_client/cdc
+
+# Run all tests with default settings
+pytest testcases/
+```
+
+**Note:** The test framework automatically configures CDC topology at startup. Default configuration uses:
+- Upstream URI: `http://10.104.17.154:19530`
+- Downstream URI: `http://10.104.17.156:19530`
+- Authentication: `root:Milvus`
+
+To customize connections, see the [Configuration](#configuration) section.
+
+### How CDC Topology Works
+
+The test framework automatically sets up CDC replication between clusters at session start:
+
+1. Creates cluster configurations with specified IDs and physical channels
+2. Establishes one-way replication: upstream → downstream
+3. Initializes the CDC connection (5-second wait period)
+
+This setup is transparent - tests will automatically use the configured topology.
+
+## Test Categories
+
+The test suite is organized into the following categories:
+
+### 1. Database Operations
+**File:** `test_database.py`
+- CREATE_DATABASE
+- DROP_DATABASE
+- ALTER_DATABASE_PROPERTIES
+- DROP_DATABASE_PROPERTIES
+
+### 2. Resource Group Operations
+**File:** `test_resource_group.py`
+- CREATE_RESOURCE_GROUP
+- DROP_RESOURCE_GROUP
+- TRANSFER_NODE
+- TRANSFER_REPLICA
+
+### 3. RBAC Operations
+**File:** `test_rbac.py`
+- CREATE_ROLE / DROP_ROLE
+- CREATE_USER / DROP_USER
+- GRANT_ROLE / REVOKE_ROLE
+- GRANT_PRIVILEGE / REVOKE_PRIVILEGE
+
+### 4. Collection DDL Operations
+**File:** `test_collection.py`
+- CREATE_COLLECTION
+- DROP_COLLECTION
+- RENAME_COLLECTION
+
+### 5. Index Operations
+**File:** `test_index.py`
+- CREATE_INDEX
+- DROP_INDEX
+
+### 6. Data Manipulation Operations
+**File:** `test_dml.py`
+- INSERT
+- DELETE
+- UPSERT
+- BULK_INSERT
+
+### 7. Collection Management Operations
+**File:** `test_collection_management.py`
+- LOAD_COLLECTION
+- RELEASE_COLLECTION
+- FLUSH
+- COMPACT
+
+### 8. Alias Operations
+**File:** `test_alias.py`
+- CREATE_ALIAS
+- DROP_ALIAS
+- ALTER_ALIAS
+
+### 9. Partition Operations
+**File:** `test_partition.py`
+- CREATE_PARTITION / DROP_PARTITION
+- LOAD_PARTITION / RELEASE_PARTITION
+- Partition data operations (INSERT, DELETE)
+
+## Configuration
+
+### Connection Parameters
+
+| Parameter | Description | Default |
+|-----------|-------------|---------|
+| `--upstream-uri` | Upstream Milvus URI | `http://10.104.17.154:19530` |
+| `--upstream-token` | Upstream authentication token | `root:Milvus` |
+| `--downstream-uri` | Downstream Milvus URI | `http://10.104.17.156:19530` |
+| `--downstream-token` | Downstream authentication token | `root:Milvus` |
+
+### CDC Topology Parameters
+
+| Parameter | Description | Default |
+|-----------|-------------|---------|
+| `--source-cluster-id` | Source cluster identifier | `cdc-test-source-0930` |
+| `--target-cluster-id` | Target cluster identifier | `cdc-test-target-0930` |
+| `--pchannel-num` | Number of physical channels | `16` |
+
+### Test Parameters
+
+| Parameter | Description | Default |
+|-----------|-------------|---------|
+| `--sync-timeout` | Sync timeout in seconds | `30` |
+
+## Usage Examples
+
+### Run Specific Test Category
+
+```bash
+# Database operation tests
+pytest testcases/test_database.py
+
+# RBAC operation tests
+pytest testcases/test_rbac.py
+
+# Data manipulation tests
+pytest testcases/test_dml.py
+```
+
+### Custom Connection Configuration
+
+```bash
+pytest testcases/ \
+  --upstream-uri http://localhost:19530 \
+  --upstream-token root:Milvus \
+  --downstream-uri http://localhost:19531 \
+  --downstream-token root:Milvus
+```
+
+### Custom Sync Timeout
+
+For slower networks or larger data volumes:
+
+```bash
+pytest testcases/test_dml.py --sync-timeout 180
+```
+
+### Custom CDC Topology
+
+Configure custom cluster IDs and channel count:
+
+```bash
+pytest testcases/ \
+  --source-cluster-id my-source \
+  --target-cluster-id my-target \
+  --pchannel-num 32
+```
+
+### Full Custom Configuration
+
+Combine all parameters:
+
+```bash
+pytest testcases/test_database.py \
+  --upstream-uri http://10.100.1.10:19530 \
+  --upstream-token root:Milvus \
+  --downstream-uri http://10.100.1.20:19530 \
+  --downstream-token root:Milvus \
+  --source-cluster-id prod-source \
+  --target-cluster-id prod-target \
+  --pchannel-num 32 \
+  --sync-timeout 180
+```
+
+### Run Specific Test Method
+
+```bash
+pytest testcases/test_database.py::TestCDCSyncDatabase::test_create_database \
+  --upstream-uri http://localhost:19530 \
+  --downstream-uri http://localhost:19531
+```
+
+## Project Structure
+
+For developers who need to understand the test organization:
+
+```
+testcases/
+├── base.py                     # Base test class and utility functions
+├── test_database.py            # Database operation tests
+├── test_rbac.py                # RBAC operation tests
+├── test_collection.py          # Collection DDL operation tests
+├── test_index.py               # Index operation tests
+├── test_dml.py                 # Data manipulation tests
+├── test_collection_management.py # Collection management tests
+├── test_alias.py               # Alias operation tests
+└── test_partition.py           # Partition operation tests
+```

tests/python_client/cdc/conftest.py

@@ -0,0 +1,479 @@
+import logging
+import time
+from concurrent.futures import ThreadPoolExecutor, wait
+
+import pytest
+from pymilvus import MilvusClient
+
+# Configure logging
+logging.basicConfig(level=logging.INFO, format="%(asctime)s - %(name)s - %(levelname)s - %(message)s")
+logger = logging.getLogger(__name__)
+
+CDC_UPDATE_REPLICATE_TIMEOUT_SECONDS = 600
+
+
+def apply_replicate_configuration(tasks, timeout=CDC_UPDATE_REPLICATE_TIMEOUT_SECONDS):
+    # Fan out in parallel: the server blocks non-primary clusters in
+    # waitUntilPrimaryChangeOrConfigurationSame until the primary's broadcast
+    # propagates via CDC, so a sequential call where the first client happens
+    # to be a replica deadlocks on the client's RPC timeout.
+    with ThreadPoolExecutor(max_workers=len(tasks)) as executor:
+        futures = [
+            executor.submit(client.update_replicate_configuration, timeout=timeout, **config)
+            for client, config in tasks
+        ]
+        wait(futures)
+    for f in futures:
+        f.result()
+
+
+def pytest_addoption(parser):
+    """Add command line options for pytest."""
+    parser.addoption(
+        "--upstream-uri",
+        action="store",
+        default="http://10.104.17.154:19530",
+        help="Upstream Milvus uri",
+    )
+    parser.addoption(
+        "--upstream-token",
+        action="store",
+        default="root:Milvus",
+        help="Upstream Milvus token",
+    )
+    parser.addoption(
+        "--downstream-uri",
+        action="store",
+        default="http://10.104.17.156:19530",
+        help="Downstream Milvus uri",
+    )
+    parser.addoption(
+        "--downstream-token",
+        action="store",
+        default="root:Milvus",
+        help="Downstream Milvus token",
+    )
+    parser.addoption("--sync-timeout", action="store", default="30", help="Sync timeout in seconds")
+    parser.addoption(
+        "--source-cluster-id",
+        action="store",
+        default="cdc-test-source-0930",
+        help="Source cluster ID for CDC topology",
+    )
+    parser.addoption(
+        "--target-cluster-id",
+        action="store",
+        default="cdc-test-target-0930",
+        help="Target cluster ID for CDC topology",
+    )
+    parser.addoption(
+        "--pchannel-num",
+        action="store",
+        default="16",
+        help="Number of physical channels for CDC",
+    )
+    parser.addoption(
+        "--request-duration",
+        action="store",
+        default="30m",
+        help="Duration for test operations (e.g., 30m, 1h, 60s)",
+    )
+    parser.addoption(
+        "--is-check",
+        action="store",
+        default="true",
+        help="Whether to assert on checker statistics",
+    )
+    parser.addoption(
+        "--milvus-ns",
+        action="store",
+        default="chaos-testing",
+        help="Kubernetes namespace for Milvus deployment",
+    )
+    parser.addoption(
+        "--import-2pc-workload",
+        action="store",
+        default="true",
+        help="Whether CDC stability suites should include Import 2PC workload",
+    )
+    parser.addoption(
+        "--import-2pc-minio-host",
+        action="store",
+        default="",
+        help="Upstream MinIO host or host:port used by Import 2PC workload",
+    )
+    parser.addoption(
+        "--import-2pc-minio-bucket",
+        action="store",
+        default="",
+        help="Upstream MinIO bucket used by Import 2PC workload",
+    )
+    parser.addoption(
+        "--import-2pc-downstream-minio-host",
+        action="store",
+        default="",
+        help="Downstream MinIO host or host:port used by CDC Import 2PC workload",
+    )
+    parser.addoption(
+        "--import-2pc-downstream-minio-bucket",
+        action="store",
+        default="",
+        help="Downstream MinIO bucket used by CDC Import 2PC workload",
+    )
+    parser.addoption(
+        "--import-2pc-rows",
+        action="store",
+        default="20",
+        help="Rows per Import 2PC checker operation",
+    )
+
+
+def _as_bool(value):
+    if isinstance(value, bool):
+        return value
+    return str(value).lower() in ("1", "true", "yes", "y")
+
+
+def _minio_endpoint(host):
+    host = (host or "").strip()
+    if not host:
+        return ""
+    if ":" in host:
+        return host
+    return f"{host}:9000"
+
+
+@pytest.fixture(scope="session")
+def upstream_client(request):
+    """Create upstream MilvusClient."""
+    uri = request.config.getoption("--upstream-uri")
+    token = request.config.getoption("--upstream-token")
+    client = MilvusClient(uri=uri, token=token)
+    yield client
+    client.close()
+
+
+@pytest.fixture(scope="session")
+def downstream_client(request):
+    """Create downstream MilvusClient."""
+    uri = request.config.getoption("--downstream-uri")
+    token = request.config.getoption("--downstream-token")
+    client = MilvusClient(uri=uri, token=token)
+    yield client
+    client.close()
+
+
+@pytest.fixture(scope="session")
+def sync_timeout(request):
+    """Get sync timeout from command line."""
+    return int(request.config.getoption("--sync-timeout"))
+
+
+@pytest.fixture(scope="session")
+def upstream_uri(request):
+    """Get upstream uri from command line."""
+    return request.config.getoption("--upstream-uri")
+
+
+@pytest.fixture(scope="session")
+def upstream_token(request):
+    """Get upstream token from command line."""
+    return request.config.getoption("--upstream-token")
+
+
+@pytest.fixture(scope="session")
+def downstream_uri(request):
+    """Get downstream uri from command line."""
+    return request.config.getoption("--downstream-uri")
+
+
+@pytest.fixture(scope="session")
+def downstream_token(request):
+    """Get downstream token from command line."""
+    return request.config.getoption("--downstream-token")
+
+
+@pytest.fixture(scope="session")
+def source_cluster_id(request):
+    """Get source cluster id from command line."""
+    return request.config.getoption("--source-cluster-id")
+
+
+@pytest.fixture(scope="session")
+def target_cluster_id(request):
+    """Get target cluster id from command line."""
+    return request.config.getoption("--target-cluster-id")
+
+
+@pytest.fixture(scope="session")
+def pchannel_num(request):
+    """Get pchannel num from command line."""
+    return int(request.config.getoption("--pchannel-num"))
+
+
+@pytest.fixture(scope="session")
+def request_duration(request):
+    """Get request duration from command line."""
+    return request.config.getoption("--request-duration")
+
+
+@pytest.fixture(scope="session")
+def is_check(request):
+    # The root tests/python_client/conftest.py registers --is_check (underscore)
+    # with type=bool, which argparse maps to the same dest (is_check) as our
+    # --is-check (hyphen). The root's bool wins in chaos runs, so accept either.
+    val = request.config.getoption("--is-check")
+    return val if isinstance(val, bool) else str(val).lower() == "true"
+
+
+@pytest.fixture(scope="session")
+def milvus_ns(request):
+    return request.config.getoption("--milvus-ns")
+
+
+@pytest.fixture(scope="session")
+def import_2pc_workload(request):
+    return _as_bool(request.config.getoption("--import-2pc-workload"))
+
+
+@pytest.fixture(scope="session")
+def import_2pc_minio_endpoint(request):
+    host = request.config.getoption("--import-2pc-minio-host") or request.config.getoption("--minio_host")
+    return _minio_endpoint(host)
+
+
+@pytest.fixture(scope="session")
+def import_2pc_minio_bucket(request):
+    return request.config.getoption("--import-2pc-minio-bucket") or request.config.getoption("--minio_bucket")
+
+
+@pytest.fixture(scope="session")
+def import_2pc_downstream_minio_endpoint(request):
+    host = (
+        request.config.getoption("--import-2pc-downstream-minio-host")
+        or request.config.getoption("--import-2pc-minio-host")
+        or request.config.getoption("--minio_host")
+    )
+    return _minio_endpoint(host)
+
+
+@pytest.fixture(scope="session")
+def import_2pc_downstream_minio_bucket(request):
+    return (
+        request.config.getoption("--import-2pc-downstream-minio-bucket")
+        or request.config.getoption("--import-2pc-minio-bucket")
+        or request.config.getoption("--minio_bucket")
+    )
+
+
+@pytest.fixture(scope="session")
+def import_2pc_rows(request):
+    return int(request.config.getoption("--import-2pc-rows"))
+
+
+@pytest.fixture(scope="session")
+def switchover_helper(request, upstream_client, downstream_client):
+    """Returns a callable that performs CDC topology switchover."""
+    upstream_uri = request.config.getoption("--upstream-uri")
+    upstream_token = request.config.getoption("--upstream-token")
+    downstream_uri = request.config.getoption("--downstream-uri")
+    downstream_token = request.config.getoption("--downstream-token")
+    pchannel_num = int(request.config.getoption("--pchannel-num"))
+    original_source = request.config.getoption("--source-cluster-id")
+    original_target = request.config.getoption("--target-cluster-id")
+
+    # Map cluster IDs to their URIs/tokens
+    cluster_map = {
+        original_source: {"uri": upstream_uri, "token": upstream_token},
+        original_target: {"uri": downstream_uri, "token": downstream_token},
+    }
+
+    def do_switchover(new_source_id, new_target_id):
+        logger.info(f"Performing switchover: {new_source_id} -> {new_target_id}")
+        config = {
+            "clusters": [
+                {
+                    "cluster_id": new_source_id,
+                    "connection_param": cluster_map[new_source_id],
+                    "pchannels": [f"{new_source_id}-rootcoord-dml_{i}" for i in range(pchannel_num)],
+                },
+                {
+                    "cluster_id": new_target_id,
+                    "connection_param": cluster_map[new_target_id],
+                    "pchannels": [f"{new_target_id}-rootcoord-dml_{i}" for i in range(pchannel_num)],
+                },
+            ],
+            "cross_cluster_topology": [{"source_cluster_id": new_source_id, "target_cluster_id": new_target_id}],
+        }
+        # Dedicated short-lived clients so switchover RPCs don't share a
+        # gRPC channel with concurrent DML on the session-scoped clients.
+        # pymilvus's connection manager closes a channel on UNAVAILABLE /
+        # STREAMING_CODE_REPLICATE_VIOLATION to trigger recovery; if a DML
+        # on the session client triggers that close while our sibling
+        # update_replicate_configuration RPC is in flight on the same
+        # channel, the latter surfaces "Cannot invoke RPC on closed
+        # channel!". Separate clients = separate channels = no race.
+        up_tmp = MilvusClient(uri=upstream_uri, token=upstream_token)
+        dn_tmp = MilvusClient(uri=downstream_uri, token=downstream_token)
+        try:
+            apply_replicate_configuration([(up_tmp, config), (dn_tmp, config)])
+        finally:
+            up_tmp.close()
+            dn_tmp.close()
+        logger.info("Switchover completed, waiting 10s for stabilization...")
+        time.sleep(10)
+
+    return do_switchover
+
+
+@pytest.fixture(scope="session")
+def kubectl_helper(milvus_ns):
+    """Helpers for pod-kill failover scenarios."""
+    import subprocess as _sp
+    import time as _time
+
+    def delete_pods(instance_label):
+        cmd = [
+            "kubectl",
+            "delete",
+            "pods",
+            "-l",
+            f"app.kubernetes.io/instance={instance_label}",
+            "-n",
+            milvus_ns,
+            "--grace-period=0",
+            "--force",
+        ]
+        result = _sp.run(cmd, capture_output=True, text=True, check=False)
+        logger.info(
+            f"[KUBECTL] delete pods {instance_label}: rc={result.returncode}, "
+            f"stdout={result.stdout!r}, stderr={result.stderr!r}"
+        )
+        return result
+
+    def wait_for_pods_ready(instance_label, timeout=300):
+        """Wait for pods matching the label to be Ready.
+
+        Two phases because right after `kubectl delete pods`, the operator
+        hasn't recreated pods yet — `kubectl wait` would error with "no
+        matching resources found". So:
+          1. Poll until at least one pod matches.
+          2. Then `kubectl wait` for Ready on the remaining time budget.
+        """
+        deadline = _time.time() + timeout
+        existence = None
+        while _time.time() < deadline:
+            existence = _sp.run(
+                [
+                    "kubectl",
+                    "get",
+                    "pods",
+                    "-l",
+                    f"app.kubernetes.io/instance={instance_label}",
+                    "-n",
+                    milvus_ns,
+                    "--no-headers",
+                ],
+                capture_output=True,
+                text=True,
+                check=False,
+            )
+            if existence.stdout.strip():
+                break
+            _time.sleep(5)
+        else:
+            raise TimeoutError(f"no pods matched {instance_label} within {timeout}s")
+
+        remaining = max(1, int(deadline - _time.time()))
+        cmd = [
+            "kubectl",
+            "wait",
+            "--for=condition=Ready",
+            "pods",
+            "-l",
+            f"app.kubernetes.io/instance={instance_label}",
+            "-n",
+            milvus_ns,
+            f"--timeout={remaining}s",
+        ]
+        result = _sp.run(cmd, capture_output=True, text=True, check=False)
+        logger.info(f"[KUBECTL] wait pods {instance_label}: rc={result.returncode}")
+        if result.returncode != 0:
+            raise TimeoutError(
+                f"pods matching {instance_label} did not become Ready within {remaining}s: "
+                f"stdout={result.stdout!r}, stderr={result.stderr!r}"
+            )
+        return result
+
+    class KubectlHelper:
+        pass
+
+    KubectlHelper.delete_pods = staticmethod(delete_pods)
+    KubectlHelper.wait_for_pods_ready = staticmethod(wait_for_pods_ready)
+
+    return KubectlHelper()
+
+
+@pytest.fixture(scope="session", autouse=True)
+def cdc_topology_setup(request, upstream_client, downstream_client):
+    """Setup CDC topology at the beginning of test session."""
+    upstream_uri = request.config.getoption("--upstream-uri")
+    downstream_uri = request.config.getoption("--downstream-uri")
+    source_cluster_id = request.config.getoption("--source-cluster-id")
+    target_cluster_id = request.config.getoption("--target-cluster-id")
+    pchannel_num = int(request.config.getoption("--pchannel-num"))
+
+    logger.info(f"Setting up CDC topology: {source_cluster_id} -> {target_cluster_id} (channels: {pchannel_num})...")
+
+    # Create CDC replication configuration
+    config = {
+        "clusters": [
+            {
+                "cluster_id": source_cluster_id,
+                "connection_param": {
+                    "uri": upstream_uri,
+                    "token": request.config.getoption("--upstream-token"),
+                },
+                "pchannels": [f"{source_cluster_id}-rootcoord-dml_{i}" for i in range(pchannel_num)],
+            },
+            {
+                "cluster_id": target_cluster_id,
+                "connection_param": {
+                    "uri": downstream_uri,
+                    "token": request.config.getoption("--downstream-token"),
+                },
+                "pchannels": [f"{target_cluster_id}-rootcoord-dml_{i}" for i in range(pchannel_num)],
+            },
+        ],
+        "cross_cluster_topology": [
+            {
+                "source_cluster_id": source_cluster_id,
+                "target_cluster_id": target_cluster_id,
+            }
+        ],
+    }
+
+    try:
+        # Dedicated clients for the control-plane update_replicate_configuration
+        # RPC, mirroring switchover_helper. Keeps the session-scoped clients'
+        # channels clean of any recovery side effects from the initial setup.
+        up_tmp = MilvusClient(uri=upstream_uri, token=request.config.getoption("--upstream-token"))
+        dn_tmp = MilvusClient(uri=downstream_uri, token=request.config.getoption("--downstream-token"))
+        try:
+            apply_replicate_configuration([(up_tmp, config), (dn_tmp, config)])
+        finally:
+            up_tmp.close()
+            dn_tmp.close()
+        logger.info("CDC topology setup completed successfully")
+
+        # Allow some time for CDC to initialize
+        time.sleep(5)
+
+    except Exception as e:
+        logger.error(f"Failed to setup CDC topology: {e}")
+        raise
+
+    yield
+
+    # Cleanup can be added here if needed
+    logger.info("CDC topology teardown completed")

tests/python_client/cdc/scripts/diff_upstream_downstream.py

@@ -0,0 +1,132 @@
+"""
+db --> collection --> partition
+
+status:
+entities num
+load status
+index status
+
+
+then load partition
+query all data
+compare result
+
+"""
+import time
+
+from loguru import logger
+import json
+import collections.abc
+from deepdiff import DeepDiff
+from pymilvus import connections, Collection, db, list_collections
+import threading
+
+
+def convert_deepdiff(diff):
+    if isinstance(diff, dict):
+        return {k: convert_deepdiff(v) for k, v in diff.items()}
+    elif isinstance(diff, collections.abc.Set):
+        return list(diff)
+    return diff
+
+
+def get_collection_info(info, db_name, c_name):
+    info[db_name][c_name] = {}
+    c = Collection(c_name)
+
+    info[db_name][c_name]['name'] = c.name
+    # logger.info(c.num_entities)
+    info[db_name][c_name]['num_entities'] = c.num_entities
+    # logger.info(c.schema)
+    info[db_name][c_name]['schema'] = len([f.name for f in c.schema.fields])
+    # logger.info(c.indexes)
+    info[db_name][c_name]['indexes'] = sorted([x.index_name for x in c.indexes])
+    # logger.info(c.partitions)
+    info[db_name][c_name]['partitions'] = sorted([p.name for p in c.partitions])
+    try:
+        replicas = len(c.get_replicas().groups)
+    except Exception as e:
+        logger.warning(e)
+        # logger.info(f"no replica for {db_name}.{c_name}")
+        replicas = 0
+    # logger.info(replicas)
+    info[db_name][c_name]['replicas'] = replicas
+    if replicas > 0:
+        try:
+            # logger.info(f"start query {db_name}.{c_name}")
+            res = c.query(expr="", output_fields=["count(*)"], timeout=60)
+            cnt = res[0]["count(*)"]
+            # logger.info(cnt)
+            info[db_name][c_name]['cnt'] = cnt
+        except Exception as e:
+            # logger.warning(f"failed to query {db_name}.{c_name}: {e}")
+            info[db_name][c_name]['cnt'] = -1
+
+
+def get_cluster_info(uri, token):
+    try:
+        connections.disconnect(alias='default')
+    except Exception as e:
+        logger.warning(e)
+    if token:
+        connections.connect(uri=uri, token=token)
+    else:
+        connections.connect(uri=uri)
+    info = {}
+    all_db = db.list_database()
+    # logger.info(all_db)
+    for db_name in all_db:
+        info[db_name] = {}
+        db.using_database(db_name)
+        all_collection = list_collections()
+        # logger.info(all_collection)
+        threads = []
+        for collection_name in all_collection:
+            t = threading.Thread(target=get_collection_info, args=(info, db_name, collection_name))
+            threads.append(t)
+            t.start()
+        for t in threads:
+            t.join()
+
+    # logger.info(json.dumps(info, indent=2))
+    return info
+
+
+if __name__ == '__main__':
+    import argparse
+
+    parser = argparse.ArgumentParser(description='connection info')
+    parser.add_argument('--upstream-uri', type=str, default='http://10.100.36.179:19530', help='milvus uri')
+    parser.add_argument('--downstream-uri', type=str, default='http://10.100.36.178:19530', help='milvus uri')
+    parser.add_argument('--upstream-token', type=str, default='root:Milvus', help='milvus token')
+    parser.add_argument('--downstream-token', type=str, default='root:Milvus', help='milvus token')
+    args = parser.parse_args()
+    diff_cnt = 0
+    diff = None
+    t0 = time.time()
+    while diff_cnt < 10:
+        upstream = get_cluster_info(args.upstream_uri, args.upstream_token)
+        downstream = get_cluster_info(args.downstream_uri, args.downstream_token)
+        diff = DeepDiff(upstream, downstream)
+        diff = convert_deepdiff(diff)
+        logger.info(f"diff: {diff}")
+        logger.info(f"diff: {json.dumps(diff, indent=2)}")
+        with open("diff.json", "w") as f:
+            json.dump(diff, f, indent=2)
+        excludedRegex = [r"root(\[\'\w+\'\])*\['num_entities'\]"]
+        diff = DeepDiff(upstream, downstream, exclude_regex_paths=excludedRegex)
+        diff = convert_deepdiff(diff)
+        logger.info(f"diff exclude num entities: {diff}")
+        logger.info(f"diff exclude num entities: {json.dumps(diff, indent=2)}")
+        diff_cnt += 1
+        if diff:
+            logger.info(f"diff exclude num entities found between upstream and downstream {json.dumps(diff, indent=2)}")
+            time.sleep(60)
+
+        else:
+            logger.info("no diff exclude num entities found between upstream and downstream")
+            break
+    tt = time.time() - t0
+    logger.info(f"total time cost: {tt:.2f} seconds")
+    if diff:
+        assert False, f"diff found between upstream and downstream {json.dumps(diff, indent=2)}"

tests/python_client/cdc/scripts/setup_cdc_topology.py

@@ -0,0 +1,173 @@
+from concurrent.futures import ThreadPoolExecutor, as_completed
+
+from pymilvus import MilvusClient
+
+# Kept in sync with cdc/conftest.py's CDC_UPDATE_REPLICATE_TIMEOUT_SECONDS.
+# Inlined because this file is executed standalone from tests/python_client/cdc/scripts/,
+# where the cdc package is not on sys.path.
+CDC_UPDATE_REPLICATE_TIMEOUT_SECONDS = 600
+
+
+def setup_cdc_topology(
+    upstream_uri,
+    downstream_uri,
+    removed_clusters_uri,
+    upstream_token,
+    downstream_token,
+    removed_clusters_token,
+    source_cluster_id,
+    target_cluster_id,
+    removed_clusters_id,
+    pchannel_num,
+):
+    print(
+        f"DEBUG: upstream_uri: {upstream_uri}, downstream_uri: {downstream_uri}, upstream_token: {upstream_token}, downstream_token: {downstream_token}, source_cluster_id: {source_cluster_id}, target_cluster_id: {target_cluster_id}, pchannel_num: {pchannel_num}"
+    )
+    upstream_client = MilvusClient(uri=upstream_uri, token=upstream_token)
+
+    # Parse comma-separated lists
+    if isinstance(downstream_uri, str) and "," in downstream_uri:
+        downstream_uris = [uri.strip() for uri in downstream_uri.split(",")]
+    else:
+        downstream_uris = [downstream_uri] if isinstance(downstream_uri, str) else downstream_uri
+
+    if isinstance(target_cluster_id, str) and "," in target_cluster_id:
+        target_cluster_ids = [cluster_id.strip() for cluster_id in target_cluster_id.split(",")]
+    else:
+        target_cluster_ids = [target_cluster_id] if isinstance(target_cluster_id, str) else target_cluster_id
+
+    if isinstance(removed_clusters_uri, str) and "," in removed_clusters_uri:
+        removed_clusters_uris = [uri.strip() for uri in removed_clusters_uri.split(",")]
+    else:
+        removed_clusters_uris = (
+            [removed_clusters_uri] if isinstance(removed_clusters_uri, str) else removed_clusters_uri
+        )
+
+    if isinstance(removed_clusters_id, str) and "," in removed_clusters_id:
+        removed_clusters_ids = [cluster_id.strip() for cluster_id in removed_clusters_id.split(",")]
+    else:
+        removed_clusters_ids = [removed_clusters_id] if isinstance(removed_clusters_id, str) else removed_clusters_id
+
+    # Ensure we have matching numbers of downstream URIs and cluster IDs
+    if len(downstream_uris) != len(target_cluster_ids):
+        raise ValueError(
+            f"Number of downstream URIs ({len(downstream_uris)}) must match number of target cluster IDs ({len(target_cluster_ids)})"
+        )
+
+    # Create downstream clients
+    downstream_clients = []
+    for downstream_uri_single in downstream_uris:
+        print(f"DEBUG: downstream_uri_single: {downstream_uri_single}, downstream_token: {downstream_token}")
+        downstream_clients.append(MilvusClient(uri=downstream_uri_single, token=downstream_token))
+
+    # Build clusters configuration
+    clusters = [
+        {
+            "cluster_id": source_cluster_id,
+            "connection_param": {"uri": upstream_uri, "token": upstream_token},
+            "pchannels": [f"{source_cluster_id}-rootcoord-dml_{i}" for i in range(pchannel_num)],
+        }
+    ]
+
+    # Add all target clusters
+    for target_id, target_uri in zip(target_cluster_ids, downstream_uris):
+        clusters.append(
+            {
+                "cluster_id": target_id,
+                "connection_param": {"uri": target_uri, "token": downstream_token},
+                "pchannels": [f"{target_id}-rootcoord-dml_{j}" for j in range(pchannel_num)],
+            }
+        )
+
+    # Build cross-cluster topology
+    cross_cluster_topology = []
+    for target_id in target_cluster_ids:
+        cross_cluster_topology.append({"source_cluster_id": source_cluster_id, "target_cluster_id": target_id})
+
+    config = {"clusters": clusters, "cross_cluster_topology": cross_cluster_topology}
+
+    # Update configuration on all clients using multi-threading
+    print(f"DEBUG: config: {config}")
+
+    def update_client_config(client, config_to_use, client_type=""):
+        try:
+            client.update_replicate_configuration(timeout=CDC_UPDATE_REPLICATE_TIMEOUT_SECONDS, **config_to_use)
+            return f"{client_type} updated successfully"
+        except Exception as e:
+            print(f"Failed to update {client_type}: {e}")
+            raise e
+
+    # Collect all update tasks
+    update_tasks = []
+
+    # Add upstream and downstream clients with normal config
+    all_clients = [upstream_client] + downstream_clients
+    for client in all_clients:
+        update_tasks.append((client, config, "Normal client"))
+
+    # Handle removed clusters - prepare them with empty topology
+    if removed_clusters_uris and removed_clusters_uris[0]:  # Check if removed_clusters_uris is not empty
+        for removed_uri, removed_id in zip(removed_clusters_uris, removed_clusters_ids):
+            if removed_uri and removed_id:  # Skip empty URIs and IDs
+                print(f"DEBUG: removed_cluster_uri: {removed_uri}, removed_clusters_token: {removed_clusters_token}")
+                removed_client = MilvusClient(uri=removed_uri, token=removed_clusters_token)
+
+                empty_config = {
+                    "clusters": [
+                        {
+                            "cluster_id": removed_id,
+                            "connection_param": {"uri": removed_uri, "token": removed_clusters_token},
+                            "pchannels": [f"{removed_id}-rootcoord-dml_{i}" for i in range(pchannel_num)],
+                        }
+                    ],
+                    "cross_cluster_topology": [],
+                }
+                print(f"DEBUG: Removing cluster {removed_id} with empty config: {empty_config}")
+                update_tasks.append((removed_client, empty_config, f"Removed cluster {removed_id}"))
+
+    # Use single ThreadPoolExecutor to update all clients concurrently
+    with ThreadPoolExecutor(max_workers=len(update_tasks)) as executor:
+        # Submit all update tasks
+        futures = [
+            executor.submit(update_client_config, client, config_to_use, client_type)
+            for client, config_to_use, client_type in update_tasks
+        ]
+
+        # Wait for all tasks to complete
+        for future in as_completed(futures):
+            try:
+                result = future.result()
+                print(f"Task completed: {result}")
+            except Exception as e:
+                print(f"Task failed with error: {e}")
+                raise e
+
+
+if __name__ == "__main__":
+    import argparse
+
+    parser = argparse.ArgumentParser(description="connection info")
+    parser.add_argument("--upstream_uri", type=str, default="10.100.36.179", help="milvus host")
+    parser.add_argument("--downstream_uri", type=str, default="10.100.36.178", help="milvus host")
+    parser.add_argument("--removed_clusters_uri", type=str, default="", help="milvus host")
+    parser.add_argument("--upstream_token", type=str, default="root:Milvus", help="milvus token")
+    parser.add_argument("--downstream_token", type=str, default="root:Milvus", help="milvus token")
+    parser.add_argument("--removed_clusters_token", type=str, default="root:Milvus", help="milvus token")
+    parser.add_argument("--source_cluster_id", type=str, default="cdc-test-source", help="source cluster id")
+    parser.add_argument("--target_cluster_id", type=str, default="cdc-test-target", help="target cluster id")
+    parser.add_argument("--removed_clusters_id", type=str, default="", help="removed clusters id")
+
+    parser.add_argument("--pchannel_num", type=int, default=16, help="pchannel num")
+    args = parser.parse_args()
+    setup_cdc_topology(
+        args.upstream_uri,
+        args.downstream_uri,
+        args.removed_clusters_uri,
+        args.upstream_token,
+        args.downstream_token,
+        args.removed_clusters_token,
+        args.source_cluster_id,
+        args.target_cluster_id,
+        args.removed_clusters_id,
+        args.pchannel_num,
+    )

tests/python_client/cdc/stablity/test_concurrent_operation.py

@@ -0,0 +1,199 @@
+import json
+import time
+from time import sleep
+
+import pytest
+from chaos import chaos_commons as cc
+from chaos import constants
+from chaos.chaos_commons import assert_statistic
+from chaos.checker import (
+    AddFieldChecker,
+    DeleteChecker,
+    FlushChecker,
+    FullTextSearchChecker,
+    GeoQueryChecker,
+    HybridSearchChecker,
+    Import2PCChecker,
+    InsertChecker,
+    JsonQueryChecker,
+    Op,
+    PhraseMatchChecker,
+    QueryChecker,
+    ResultAnalyzer,
+    SearchChecker,
+    TextMatchChecker,
+    UpsertChecker,
+)
+from common import common_func as cf
+from common.common_type import CaseLabel
+from common.milvus_sys import MilvusSys
+from delayed_assert import assert_expectations
+from pymilvus import DataType, FunctionType, connections
+from utils.util_k8s import get_milvus_instance_name, wait_pods_ready
+from utils.util_log import test_log as log
+
+_VECTOR_DTYPES = {
+    DataType.FLOAT_VECTOR,
+    DataType.FLOAT16_VECTOR,
+    DataType.BFLOAT16_VECTOR,
+    DataType.BINARY_VECTOR,
+    DataType.SPARSE_FLOAT_VECTOR,
+    DataType.INT8_VECTOR,
+}
+
+
+def _build_checker_schema(dim=8):
+    """Build the shared all-datatype schema, stripped for the 2.6-latest image.
+
+    The chaos-test image used by milvus_cdc_chaos_test/verify_test rejects
+    two things the shared gen_all_datatype_collection_schema includes by
+    default:
+      - FunctionType.MINHASH (error: "check function params with unknown
+        function type")
+      - nullable=True on FLOAT_VECTOR (error: "vector type not support null")
+
+    Drop the MinHash function and its output field, and force nullable=False
+    on every vector field so the server accepts the schema.
+    """
+    schema = cf.gen_all_datatype_collection_schema(dim=dim)
+    schema.functions[:] = [f for f in schema.functions if f.type != FunctionType.MINHASH]
+    schema.fields[:] = [f for f in schema.fields if f.name != "minhash_emb"]
+    for f in schema.fields:
+        if f.dtype in _VECTOR_DTYPES:
+            f.nullable = False
+    return schema
+
+
+def get_all_collections():
+    try:
+        with open("/tmp/ci_logs/chaos_test_all_collections.json") as f:
+            data = json.load(f)
+            all_collections = data["all"]
+    except Exception as e:
+        log.warning(f"get_all_collections error: {e}")
+        return [None]
+    return all_collections
+
+
+class TestBase:
+    expect_create = constants.SUCC
+    expect_insert = constants.SUCC
+    expect_flush = constants.SUCC
+    expect_compact = constants.SUCC
+    expect_search = constants.SUCC
+    expect_query = constants.SUCC
+    host = "127.0.0.1"
+    port = 19530
+    _chaos_config = None
+    health_checkers = {}
+
+
+class TestOperations(TestBase):
+    @pytest.fixture(scope="function", autouse=True)
+    def connection(
+        self,
+        upstream_uri,
+        upstream_token,
+        downstream_uri,
+        downstream_token,
+        milvus_ns,
+        import_2pc_workload,
+        import_2pc_minio_endpoint,
+        import_2pc_minio_bucket,
+        import_2pc_downstream_minio_endpoint,
+        import_2pc_downstream_minio_bucket,
+        import_2pc_rows,
+    ):
+        connections.connect("default", uri=upstream_uri, token=upstream_token)
+        if connections.has_connection("default") is False:
+            raise Exception("no connections")
+        log.info("connect to milvus successfully")
+        self.milvus_sys = MilvusSys(alias="default")
+        self.milvus_ns = milvus_ns
+        self.release_name = get_milvus_instance_name(self.milvus_ns, milvus_sys=self.milvus_sys)
+        cf.param_info.param_uri = upstream_uri
+        cf.param_info.param_token = upstream_token
+        cf.param_info.param_bucket_name = import_2pc_minio_bucket
+        self.upstream_uri = upstream_uri
+        self.upstream_token = upstream_token
+        self.downstream_uri = downstream_uri
+        self.downstream_token = downstream_token
+        self.import_2pc_workload = import_2pc_workload
+        self.import_2pc_minio_endpoint = import_2pc_minio_endpoint
+        self.import_2pc_minio_bucket = import_2pc_minio_bucket
+        self.import_2pc_downstream_minio_endpoint = import_2pc_downstream_minio_endpoint
+        self.import_2pc_downstream_minio_bucket = import_2pc_downstream_minio_bucket
+        self.import_2pc_rows = import_2pc_rows
+
+    def init_health_checkers(self, collection_name=None):
+        c_name = collection_name
+        schema = _build_checker_schema()
+        checkers = {
+            Op.insert: InsertChecker(collection_name=c_name, schema=schema),
+            Op.upsert: UpsertChecker(collection_name=c_name, schema=schema),
+            Op.flush: FlushChecker(collection_name=c_name, schema=schema),
+            Op.search: SearchChecker(collection_name=c_name, schema=schema),
+            Op.full_text_search: FullTextSearchChecker(collection_name=c_name, schema=schema),
+            Op.hybrid_search: HybridSearchChecker(collection_name=c_name, schema=schema),
+            Op.query: QueryChecker(collection_name=c_name, schema=schema),
+            Op.text_match: TextMatchChecker(collection_name=c_name, schema=schema),
+            Op.phrase_match: PhraseMatchChecker(collection_name=c_name, schema=schema),
+            Op.json_query: JsonQueryChecker(collection_name=c_name, schema=schema),
+            Op.geo_query: GeoQueryChecker(collection_name=c_name, schema=schema),
+            Op.delete: DeleteChecker(collection_name=c_name, schema=schema),
+            Op.add_field: AddFieldChecker(collection_name=c_name, schema=schema),
+        }
+        if self.import_2pc_workload:
+            checkers[Op.import_2pc] = Import2PCChecker(
+                collection_name=c_name,
+                schema=schema,
+                rows_per_import=self.import_2pc_rows,
+                minio_endpoint=self.import_2pc_minio_endpoint,
+                bucket_name=self.import_2pc_minio_bucket,
+                uri=self.upstream_uri,
+                token=self.upstream_token,
+                downstream_uri=self.downstream_uri,
+                downstream_token=self.downstream_token,
+                downstream_minio_endpoint=self.import_2pc_downstream_minio_endpoint,
+                downstream_bucket_name=self.import_2pc_downstream_minio_bucket,
+                strict_count=False,
+                pk_start=-int(time.time() * 100000),
+            )
+        log.info(f"init_health_checkers: {checkers}")
+        self.health_checkers = checkers
+
+    @pytest.fixture(scope="function", params=get_all_collections())
+    def collection_name(self, request):
+        if request.param == [] or request.param == "":
+            pytest.skip("The collection name is invalid")
+        yield request.param
+
+    @pytest.mark.tags(CaseLabel.CDC)
+    def test_operations(self, request_duration, is_check, collection_name):
+        # start the monitor threads to check the milvus ops
+        log.info("*********************Test Start**********************")
+        log.info(connections.get_connection_addr("default"))
+        # event_records = EventRecords()
+        c_name = collection_name if collection_name else cf.gen_unique_str("Checker_")
+        # event_records.insert("init_health_checkers", "start")
+        self.init_health_checkers(collection_name=c_name)
+        # event_records.insert("init_health_checkers", "finished")
+        cc.start_monitor_threads(self.health_checkers)
+        log.info("*********************Load Start**********************")
+        request_duration = request_duration.replace("h", "*3600+").replace("m", "*60+").replace("s", "")
+        if request_duration[-1] == "+":
+            request_duration = request_duration[:-1]
+        request_duration = eval(request_duration)
+        for i in range(10):
+            sleep(request_duration // 10)
+            for k, v in self.health_checkers.items():
+                v.check_result()
+                # log.info(v.check_result())
+        wait_pods_ready(self.milvus_ns, f"app.kubernetes.io/instance={self.release_name}")
+        time.sleep(60)
+        ra = ResultAnalyzer()
+        ra.get_stage_success_rate()
+        if is_check:
+            assert_statistic(self.health_checkers)
+            assert_expectations()
+        log.info("*********************Chaos Test Completed**********************")

tests/python_client/cdc/stablity/test_single_request_operation.py

@@ -0,0 +1,177 @@
+import time
+from time import sleep
+
+import pymilvus
+import pytest
+from chaos import chaos_commons as cc
+from chaos import constants
+from chaos.chaos_commons import assert_statistic
+from chaos.checker import (
+    AddFieldChecker,
+    AlterCollectionChecker,
+    CollectionCreateChecker,
+    CollectionDropChecker,
+    CollectionRenameChecker,
+    DeleteChecker,
+    EventRecords,
+    FlushChecker,
+    FullTextSearchChecker,
+    GeoQueryChecker,
+    HybridSearchChecker,
+    Import2PCChecker,
+    IndexCreateChecker,
+    InsertChecker,
+    JsonQueryChecker,
+    Op,
+    PartialUpdateChecker,
+    PhraseMatchChecker,
+    QueryChecker,
+    ResultAnalyzer,
+    SearchChecker,
+    TextMatchChecker,
+    UpsertChecker,
+)
+from common import common_func as cf
+from common.common_type import CaseLabel
+from common.milvus_sys import MilvusSys
+from delayed_assert import assert_expectations
+from pymilvus import connections, utility
+from utils.util_k8s import get_milvus_instance_name, wait_pods_ready
+from utils.util_log import test_log as log
+
+
+def _build_checker_schema(dim=8, enable_struct_array_field=True):
+    return cf.gen_all_datatype_collection_schema(dim=dim, enable_struct_array_field=enable_struct_array_field)
+
+
+class TestBase:
+    expect_create = constants.SUCC
+    expect_insert = constants.SUCC
+    expect_flush = constants.SUCC
+    expect_index = constants.SUCC
+    expect_search = constants.SUCC
+    expect_query = constants.SUCC
+    host = "127.0.0.1"
+    port = 19530
+    _chaos_config = None
+    health_checkers = {}
+
+
+class TestOperations(TestBase):
+    @pytest.fixture(scope="function", autouse=True)
+    def connection(
+        self,
+        upstream_uri,
+        upstream_token,
+        downstream_uri,
+        downstream_token,
+        milvus_ns,
+        import_2pc_workload,
+        import_2pc_minio_endpoint,
+        import_2pc_minio_bucket,
+        import_2pc_downstream_minio_endpoint,
+        import_2pc_downstream_minio_bucket,
+        import_2pc_rows,
+    ):
+        connections.connect("default", uri=upstream_uri, token=upstream_token)
+        if connections.has_connection("default") is False:
+            raise Exception("no connections")
+        log.info("connect to milvus successfully")
+        pymilvus_version = pymilvus.__version__
+        server_version = utility.get_server_version()
+        log.info(f"server version: {server_version}")
+        log.info(f"pymilvus version: {pymilvus_version}")
+        self.milvus_sys = MilvusSys(alias="default")
+        self.milvus_ns = milvus_ns
+        self.release_name = get_milvus_instance_name(self.milvus_ns, milvus_sys=self.milvus_sys)
+        cf.param_info.param_uri = upstream_uri
+        cf.param_info.param_token = upstream_token
+        cf.param_info.param_bucket_name = import_2pc_minio_bucket
+        self.upstream_uri = upstream_uri
+        self.upstream_token = upstream_token
+        self.downstream_uri = downstream_uri
+        self.downstream_token = downstream_token
+        self.import_2pc_workload = import_2pc_workload
+        self.import_2pc_minio_endpoint = import_2pc_minio_endpoint
+        self.import_2pc_minio_bucket = import_2pc_minio_bucket
+        self.import_2pc_downstream_minio_endpoint = import_2pc_downstream_minio_endpoint
+        self.import_2pc_downstream_minio_bucket = import_2pc_downstream_minio_bucket
+        self.import_2pc_rows = import_2pc_rows
+
+    def init_health_checkers(self, collection_name=None):
+        c_name = collection_name
+        schema = _build_checker_schema()
+        partial_update_schema = _build_checker_schema(enable_struct_array_field=False)
+        checkers = {
+            Op.create: CollectionCreateChecker(collection_name=c_name, schema=schema),
+            Op.insert: InsertChecker(collection_name=c_name, schema=schema),
+            Op.upsert: UpsertChecker(collection_name=c_name, schema=schema),
+            Op.partial_update: PartialUpdateChecker(schema=partial_update_schema),
+            Op.flush: FlushChecker(collection_name=c_name, schema=schema),
+            Op.index: IndexCreateChecker(collection_name=c_name, schema=schema),
+            Op.search: SearchChecker(collection_name=c_name, schema=schema),
+            Op.full_text_search: FullTextSearchChecker(collection_name=c_name, schema=schema),
+            Op.hybrid_search: HybridSearchChecker(collection_name=c_name, schema=schema),
+            Op.query: QueryChecker(collection_name=c_name, schema=schema),
+            Op.text_match: TextMatchChecker(collection_name=c_name, schema=schema),
+            Op.phrase_match: PhraseMatchChecker(collection_name=c_name, schema=schema),
+            Op.json_query: JsonQueryChecker(collection_name=c_name, schema=schema),
+            Op.geo_query: GeoQueryChecker(collection_name=c_name, schema=schema),
+            Op.delete: DeleteChecker(collection_name=c_name, schema=schema),
+            Op.drop: CollectionDropChecker(collection_name=c_name, schema=schema),
+            Op.alter_collection: AlterCollectionChecker(collection_name=c_name, schema=schema),
+            Op.add_field: AddFieldChecker(collection_name=c_name, schema=schema),
+            Op.rename_collection: CollectionRenameChecker(collection_name=c_name, schema=schema),
+        }
+        if self.import_2pc_workload:
+            checkers[Op.import_2pc] = Import2PCChecker(
+                collection_name=cf.gen_unique_str("Import2PCChecker_"),
+                rows_per_import=self.import_2pc_rows,
+                minio_endpoint=self.import_2pc_minio_endpoint,
+                bucket_name=self.import_2pc_minio_bucket,
+                uri=self.upstream_uri,
+                token=self.upstream_token,
+                downstream_uri=self.downstream_uri,
+                downstream_token=self.downstream_token,
+                downstream_minio_endpoint=self.import_2pc_downstream_minio_endpoint,
+                downstream_bucket_name=self.import_2pc_downstream_minio_bucket,
+            )
+        self.health_checkers = checkers
+
+    @pytest.mark.tags(CaseLabel.CDC)
+    def test_operations(self, request_duration, is_check):
+        # start the monitor threads to check the milvus ops
+        log.info("*********************Test Start**********************")
+        log.info(connections.get_connection_addr("default"))
+        event_records = EventRecords()
+        c_name = None
+        event_records.insert("init_health_checkers", "start")
+        self.init_health_checkers(collection_name=c_name)
+        event_records.insert("init_health_checkers", "finished")
+        tasks = cc.start_monitor_threads(self.health_checkers)
+        log.info("*********************Load Start**********************")
+        # wait request_duration
+        request_duration = request_duration.replace("h", "*3600+").replace("m", "*60+").replace("s", "")
+        if request_duration[-1] == "+":
+            request_duration = request_duration[:-1]
+        request_duration = eval(request_duration)
+        for i in range(10):
+            sleep(request_duration // 10)
+            # add an event so that the chaos can start to apply
+            if i == 3:
+                event_records.insert("init_chaos", "ready")
+            for k, v in self.health_checkers.items():
+                v.check_result()
+        if is_check:
+            assert_statistic(self.health_checkers, succ_rate_threshold=0.98)
+            assert_expectations()
+        # wait all pod ready
+        wait_pods_ready(self.milvus_ns, f"app.kubernetes.io/instance={self.release_name}")
+        time.sleep(60)
+        cc.check_thread_status(tasks)
+        for k, v in self.health_checkers.items():
+            v.pause()
+        ra = ResultAnalyzer()
+        ra.get_stage_success_rate()
+        ra.show_result_table()
+        log.info("*********************Chaos Test Completed**********************")

tests/python_client/cdc/testcases/__init__.py

@@ -0,0 +1,6 @@
+"""
+CDC sync test cases module.
+
+This module contains organized test cases for CDC synchronization testing.
+Each test file focuses on a specific category of operations.
+"""

tests/python_client/cdc/testcases/test_alias.py

@@ -0,0 +1,390 @@
+"""
+CDC sync tests for alias operations.
+"""
+
+import time
+import pytest
+from common.common_type import CaseLabel
+from .base import TestCDCSyncBase, logger
+
+
+@pytest.mark.tags(CaseLabel.CDC)
+class TestCDCSyncAlias(TestCDCSyncBase):
+    """Test CDC sync for alias operations."""
+
+    def setup_method(self):
+        """Setup for each test method."""
+        logger.info("Setting up test method")
+        self.resources_to_cleanup = []
+
+    def teardown_method(self):
+        """Cleanup after each test method - only cleanup upstream, downstream will sync."""
+        logger.info("Starting teardown method")
+        upstream_client = getattr(self, "_upstream_client", None)
+
+        if upstream_client:
+            logger.info(f"Cleaning up {len(self.resources_to_cleanup)} resources")
+
+            # First pass: cleanup aliases (must be done before collections)
+            logger.info("Cleaning up aliases first")
+            for resource_type, resource_name in self.resources_to_cleanup:
+                if resource_type == "alias":
+                    logger.info(f"Cleaning up alias: {resource_name}")
+                    try:
+                        upstream_client.drop_alias(resource_name)
+                        logger.info(f"Successfully dropped alias: {resource_name}")
+                    except Exception as e:
+                        logger.warning(f"Failed to drop alias {resource_name}: {e}")
+
+            # Second pass: cleanup collections (after aliases are removed)
+            logger.info("Cleaning up collections after aliases")
+            for resource_type, resource_name in self.resources_to_cleanup:
+                if resource_type == "collection":
+                    logger.info(f"Cleaning up collection: {resource_name}")
+                    self.cleanup_collection(upstream_client, resource_name)
+
+            logger.info("Waiting 1 second for cleanup to sync to downstream")
+            time.sleep(1)  # Allow cleanup to sync to downstream
+        else:
+            logger.info("No upstream client found, skipping cleanup")
+        logger.info("Teardown method completed")
+
+    def test_create_alias(self, upstream_client, downstream_client, sync_timeout):
+        """Test CREATE_ALIAS operation sync."""
+        logger.info("=== Starting test_create_alias ===")
+        # Store upstream client for teardown
+        self._upstream_client = upstream_client
+
+        collection_name = self.gen_unique_name("test_col_alias_create")
+        alias_name = self.gen_unique_name("test_alias_create")
+        logger.info(f"Generated collection name: {collection_name}")
+        logger.info(f"Generated alias name: {alias_name}")
+        self.resources_to_cleanup.append(("collection", collection_name))
+        self.resources_to_cleanup.append(("alias", alias_name))
+
+        # Initial cleanup
+        logger.info(f"Performing initial cleanup for collection: {collection_name}")
+        self.cleanup_collection(upstream_client, collection_name)
+
+        # Create collection
+        logger.info(f"Creating collection: {collection_name}")
+        upstream_client.create_collection(
+            collection_name=collection_name,
+            schema=self.create_default_schema(upstream_client),
+        )
+        logger.info(f"Collection {collection_name} created in upstream")
+
+        # Wait for creation to sync
+        logger.info(
+            f"Waiting for collection {collection_name} to sync to downstream (timeout: {sync_timeout}s)"
+        )
+
+        def check_create():
+            has_collection = downstream_client.has_collection(collection_name)
+            if has_collection:
+                logger.info(f"Collection {collection_name} found in downstream")
+            return has_collection
+
+        assert self.wait_for_sync(
+            check_create, sync_timeout, f"create collection {collection_name}"
+        )
+
+        # Create alias
+        logger.info(f"Creating alias {alias_name} for collection {collection_name}")
+        upstream_client.create_alias(collection_name, alias_name)
+        logger.info(f"Alias {alias_name} created in upstream")
+
+        # Verify alias exists in upstream
+        logger.info("Verifying alias exists in upstream")
+        upstream_aliases_result = upstream_client.list_aliases()
+        logger.info(f"Upstream aliases result: {upstream_aliases_result}")
+        upstream_aliases = upstream_aliases_result.get("aliases", [])
+        assert alias_name in upstream_aliases
+        logger.info(f"Confirmed alias {alias_name} exists in upstream")
+
+        # Verify alias points to correct collection using describe_alias
+        logger.info(
+            f"Verifying alias {alias_name} points to collection {collection_name}"
+        )
+        alias_desc = upstream_client.describe_alias(alias_name)
+        logger.info(f"Alias description: {alias_desc}")
+        assert alias_desc.get("collection_name") == collection_name
+        logger.info(
+            f"Confirmed alias {alias_name} correctly points to {collection_name}"
+        )
+
+        # Wait for alias sync to downstream
+        logger.info(
+            f"Waiting for alias {alias_name} to sync to downstream (timeout: {sync_timeout}s)"
+        )
+
+        def check_alias():
+            try:
+                downstream_aliases_result = downstream_client.list_aliases()
+                logger.info(f"Downstream aliases result: {downstream_aliases_result}")
+                downstream_aliases = downstream_aliases_result.get("aliases", [])
+                if alias_name in downstream_aliases:
+                    logger.info(f"Alias {alias_name} found in downstream")
+                    # Also verify alias points to correct collection in downstream
+                    try:
+                        downstream_alias_desc = downstream_client.describe_alias(
+                            alias_name
+                        )
+                        logger.info(
+                            f"Downstream alias description: {downstream_alias_desc}"
+                        )
+                        if (
+                            downstream_alias_desc.get("collection_name")
+                            == collection_name
+                        ):
+                            logger.info(
+                                f"Downstream alias {alias_name} correctly points to {collection_name}"
+                            )
+                            return True
+                        else:
+                            logger.warning(
+                                f"Downstream alias {alias_name} points to wrong collection: {downstream_alias_desc.get('collection_name')}"
+                            )
+                            return False
+                    except Exception as desc_e:
+                        logger.warning(f"Error describing downstream alias: {desc_e}")
+                        return False
+                return False
+            except Exception as e:
+                logger.warning(f"Error checking downstream aliases: {e}")
+                return False
+
+        assert self.wait_for_sync(
+            check_alias, sync_timeout, f"create alias {alias_name}"
+        )
+        logger.info("=== test_create_alias completed successfully ===")
+
+    def test_drop_alias(self, upstream_client, downstream_client, sync_timeout):
+        """Test DROP_ALIAS operation sync."""
+        logger.info("=== Starting test_drop_alias ===")
+        # Store upstream client for teardown
+        self._upstream_client = upstream_client
+
+        collection_name = self.gen_unique_name("test_col_alias_drop")
+        alias_name = self.gen_unique_name("test_alias_drop")
+        logger.info(f"Generated collection name: {collection_name}")
+        logger.info(f"Generated alias name: {alias_name}")
+        self.resources_to_cleanup.append(("collection", collection_name))
+        self.resources_to_cleanup.append(("alias", alias_name))
+
+        # Initial cleanup
+        logger.info(f"Performing initial cleanup for collection: {collection_name}")
+        self.cleanup_collection(upstream_client, collection_name)
+
+        # Create collection and alias
+        logger.info(f"Creating collection: {collection_name}")
+        upstream_client.create_collection(
+            collection_name=collection_name,
+            schema=self.create_default_schema(upstream_client),
+        )
+        logger.info(f"Collection {collection_name} created in upstream")
+
+        logger.info(f"Creating alias {alias_name} for collection {collection_name}")
+        upstream_client.create_alias(collection_name, alias_name)
+        logger.info(f"Alias {alias_name} created in upstream")
+
+        # Wait for setup to sync
+        logger.info(
+            f"Waiting for collection and alias setup to sync to downstream (timeout: {sync_timeout}s)"
+        )
+
+        def check_setup():
+            try:
+                has_collection = downstream_client.has_collection(collection_name)
+                downstream_aliases_result = downstream_client.list_aliases()
+                downstream_aliases = downstream_aliases_result.get("aliases", [])
+                has_alias = alias_name in downstream_aliases
+                logger.info(
+                    f"Downstream - has_collection: {has_collection}, has_alias: {has_alias}"
+                )
+                return has_collection and has_alias
+            except Exception as e:
+                logger.warning(f"Error checking downstream setup: {e}")
+                return False
+
+        assert self.wait_for_sync(
+            check_setup, sync_timeout, f"setup collection and alias {collection_name}"
+        )
+
+        # Drop alias
+        logger.info(f"Dropping alias {alias_name} from upstream")
+        upstream_client.drop_alias(alias_name)
+        logger.info(f"Alias {alias_name} dropped from upstream")
+
+        # Verify alias is dropped in upstream
+        logger.info("Verifying alias is dropped in upstream")
+        upstream_aliases_result = upstream_client.list_aliases()
+        logger.info(f"Upstream aliases result after drop: {upstream_aliases_result}")
+        upstream_aliases = upstream_aliases_result.get("aliases", [])
+        assert alias_name not in upstream_aliases
+        logger.info(f"Confirmed alias {alias_name} is dropped from upstream")
+
+        # Wait for drop to sync to downstream
+        logger.info(
+            f"Waiting for alias drop to sync to downstream (timeout: {sync_timeout}s)"
+        )
+
+        def check_drop():
+            try:
+                downstream_aliases_result = downstream_client.list_aliases()
+                logger.info(f"Downstream aliases result: {downstream_aliases_result}")
+                downstream_aliases = downstream_aliases_result.get("aliases", [])
+                is_dropped = alias_name not in downstream_aliases
+                if is_dropped:
+                    logger.info(
+                        f"Alias {alias_name} successfully dropped from downstream"
+                    )
+                return is_dropped
+            except Exception as e:
+                logger.warning(f"Error checking downstream aliases during drop: {e}")
+                return True  # If error, assume alias is dropped
+
+        assert self.wait_for_sync(check_drop, sync_timeout, f"drop alias {alias_name}")
+        logger.info("=== test_drop_alias completed successfully ===")
+
+    def test_alter_alias(self, upstream_client, downstream_client, sync_timeout):
+        """Test ALTER_ALIAS operation sync."""
+        logger.info("=== Starting test_alter_alias ===")
+        # Store upstream client for teardown
+        self._upstream_client = upstream_client
+
+        old_collection = self.gen_unique_name("test_col_alias_old")
+        new_collection = self.gen_unique_name("test_col_alias_new")
+        alias_name = self.gen_unique_name("test_alias_alter")
+        logger.info(f"Generated old collection name: {old_collection}")
+        logger.info(f"Generated new collection name: {new_collection}")
+        logger.info(f"Generated alias name: {alias_name}")
+        self.resources_to_cleanup.append(("collection", old_collection))
+        self.resources_to_cleanup.append(("collection", new_collection))
+        self.resources_to_cleanup.append(("alias", alias_name))
+
+        # Initial cleanup
+        logger.info(
+            f"Performing initial cleanup for collections: {old_collection}, {new_collection}"
+        )
+        self.cleanup_collection(upstream_client, old_collection)
+        self.cleanup_collection(upstream_client, new_collection)
+
+        # Create both collections
+        logger.info(f"Creating old collection: {old_collection}")
+        upstream_client.create_collection(
+            collection_name=old_collection,
+            schema=self.create_default_schema(upstream_client),
+        )
+        logger.info(f"Old collection {old_collection} created in upstream")
+
+        logger.info(f"Creating new collection: {new_collection}")
+        upstream_client.create_collection(
+            collection_name=new_collection,
+            schema=self.create_default_schema(upstream_client),
+        )
+        logger.info(f"New collection {new_collection} created in upstream")
+
+        # Create alias pointing to old collection
+        logger.info(
+            f"Creating alias {alias_name} pointing to old collection {old_collection}"
+        )
+        upstream_client.create_alias(old_collection, alias_name)
+        logger.info(
+            f"Alias {alias_name} created in upstream pointing to {old_collection}"
+        )
+
+        # Wait for setup to sync
+        logger.info(
+            f"Waiting for collections and alias setup to sync to downstream (timeout: {sync_timeout}s)"
+        )
+
+        def check_setup():
+            try:
+                has_old = downstream_client.has_collection(old_collection)
+                has_new = downstream_client.has_collection(new_collection)
+                downstream_aliases_result = downstream_client.list_aliases()
+                downstream_aliases = downstream_aliases_result.get("aliases", [])
+                has_alias = alias_name in downstream_aliases
+                logger.info(
+                    f"Downstream - has_old: {has_old}, has_new: {has_new}, has_alias: {has_alias}"
+                )
+                return has_old and has_new and has_alias
+            except Exception as e:
+                logger.warning(f"Error checking downstream setup: {e}")
+                return False
+
+        assert self.wait_for_sync(
+            check_setup, sync_timeout, "setup collections and alias"
+        )
+
+        # Alter alias to point to new collection
+        logger.info(
+            f"Altering alias {alias_name} to point to new collection {new_collection}"
+        )
+        upstream_client.alter_alias(new_collection, alias_name)
+        logger.info(
+            f"Alias {alias_name} altered in upstream to point to {new_collection}"
+        )
+
+        # Verify alias alteration in upstream
+        logger.info(
+            f"Verifying alias {alias_name} now points to {new_collection} in upstream"
+        )
+        upstream_alias_desc = upstream_client.describe_alias(alias_name)
+        logger.info(f"Upstream alias description after alter: {upstream_alias_desc}")
+        assert upstream_alias_desc.get("collection_name") == new_collection
+        logger.info(
+            f"Confirmed upstream alias {alias_name} now points to {new_collection}"
+        )
+
+        # Wait for alter to sync
+        logger.info(
+            f"Waiting for alias alter to sync to downstream (timeout: {sync_timeout}s)"
+        )
+
+        def check_alter():
+            try:
+                # Check if alias still exists and points to correct collection after alter
+                downstream_aliases_result = downstream_client.list_aliases()
+                logger.info(
+                    f"Downstream aliases result after alter: {downstream_aliases_result}"
+                )
+                downstream_aliases = downstream_aliases_result.get("aliases", [])
+                if alias_name in downstream_aliases:
+                    logger.info(f"Alias {alias_name} found in downstream after alter")
+                    # Verify alias points to new collection
+                    try:
+                        downstream_alias_desc = downstream_client.describe_alias(
+                            alias_name
+                        )
+                        logger.info(
+                            f"Downstream alias description after alter: {downstream_alias_desc}"
+                        )
+                        if (
+                            downstream_alias_desc.get("collection_name")
+                            == new_collection
+                        ):
+                            logger.info(
+                                f"Downstream alias {alias_name} correctly points to {new_collection}"
+                            )
+                            return True
+                        else:
+                            logger.warning(
+                                f"Downstream alias {alias_name} still points to old collection: {downstream_alias_desc.get('collection_name')}"
+                            )
+                            return False
+                    except Exception as desc_e:
+                        logger.warning(
+                            f"Error describing downstream alias after alter: {desc_e}"
+                        )
+                        return False
+                return False
+            except Exception as e:
+                logger.warning(f"Error checking downstream aliases after alter: {e}")
+                return False
+
+        assert self.wait_for_sync(
+            check_alter, sync_timeout, f"alter alias {alias_name}"
+        )
+        logger.info("=== test_alter_alias completed successfully ===")

tests/python_client/cdc/testcases/test_collection.py

@@ -0,0 +1,907 @@
+"""
+CDC sync tests for collection DDL operations.
+"""
+
+import time
+import pytest
+from pymilvus import DataType, Collection
+from common.common_type import CaseLabel
+from .base import TestCDCSyncBase, logger
+
+
+@pytest.mark.tags(CaseLabel.CDC)
+class TestCDCSyncCollectionDDL(TestCDCSyncBase):
+    """Test CDC sync for collection DDL operations."""
+
+    def setup_method(self):
+        """Setup for each test method."""
+        self.resources_to_cleanup = []
+
+    def teardown_method(self):
+        """Cleanup after each test method - only cleanup upstream, downstream will sync."""
+        upstream_client = getattr(self, "_upstream_client", None)
+
+        if upstream_client:
+            for resource_type, resource_name in self.resources_to_cleanup:
+                if resource_type == "collection":
+                    self.cleanup_collection(upstream_client, resource_name)
+
+            time.sleep(1)  # Allow cleanup to sync to downstream
+
+    def test_create_collection(self, upstream_client, downstream_client, sync_timeout):
+        """Test CREATE_COLLECTION operation sync."""
+        start_time = time.time()
+        collection_name = self.gen_unique_name("test_col_create")
+
+        # Log test start
+        self.log_test_start(
+            "test_create_collection", "CREATE_COLLECTION", collection_name
+        )
+
+        # Store upstream client for teardown
+        self._upstream_client = upstream_client
+        self.resources_to_cleanup.append(("collection", collection_name))
+
+        try:
+            # Initial cleanup
+            self.cleanup_collection(upstream_client, collection_name)
+
+            # Log operation
+            self.log_operation(
+                "CREATE_COLLECTION", "collection", collection_name, "upstream"
+            )
+
+            # Create collection in upstream
+            schema = self.create_default_schema(upstream_client)
+            logger.info(f"[SCHEMA] Collection schema: {schema}")
+
+            upstream_client.create_collection(
+                collection_name=collection_name, schema=schema
+            )
+
+            # Verify upstream creation
+            upstream_exists = upstream_client.has_collection(collection_name)
+            self.log_resource_state(
+                "collection",
+                collection_name,
+                "exists" if upstream_exists else "missing",
+                "upstream",
+            )
+            assert upstream_exists, (
+                f"Collection {collection_name} not created in upstream"
+            )
+
+            # Log sync verification start
+            self.log_sync_verification(
+                "CREATE_COLLECTION", collection_name, "exists in downstream"
+            )
+
+            # Wait for sync to downstream
+            def check_sync():
+                exists = downstream_client.has_collection(collection_name)
+                if exists:
+                    self.log_resource_state(
+                        "collection",
+                        collection_name,
+                        "exists",
+                        "downstream",
+                        "Sync confirmed",
+                    )
+                return exists
+
+            sync_success = self.wait_for_sync(
+                check_sync, sync_timeout, f"create collection {collection_name}"
+            )
+            assert sync_success, (
+                f"Collection {collection_name} failed to sync to downstream"
+            )
+
+            # Log test success
+            duration = time.time() - start_time
+            self.log_test_end("test_create_collection", True, duration)
+
+        except Exception as e:
+            duration = time.time() - start_time
+            logger.error(f"[ERROR] Test failed with error: {e}")
+            self.log_test_end("test_create_collection", False, duration)
+            raise
+
+    def test_drop_collection(self, upstream_client, downstream_client, sync_timeout):
+        """Test DROP_COLLECTION operation sync."""
+        start_time = time.time()
+        collection_name = self.gen_unique_name("test_col_drop")
+
+        # Log test start
+        self.log_test_start("test_drop_collection", "DROP_COLLECTION", collection_name)
+
+        # Store upstream client for teardown
+        self._upstream_client = upstream_client
+        self.resources_to_cleanup.append(("collection", collection_name))
+
+        try:
+            # Initial cleanup
+            self.cleanup_collection(upstream_client, collection_name)
+
+            # Create collection first
+            self.log_operation(
+                "CREATE_COLLECTION", "collection", collection_name, "upstream"
+            )
+            upstream_client.create_collection(
+                collection_name=collection_name,
+                schema=self.create_default_schema(upstream_client),
+            )
+
+            # Wait for creation to sync
+            def check_create():
+                return downstream_client.has_collection(collection_name)
+
+            assert self.wait_for_sync(
+                check_create, sync_timeout, f"create collection {collection_name}"
+            )
+
+            # Drop collection in upstream
+            self.log_operation(
+                "DROP_COLLECTION", "collection", collection_name, "upstream"
+            )
+            upstream_client.drop_collection(collection_name)
+
+            # Verify upstream drop
+            upstream_exists = upstream_client.has_collection(collection_name)
+            self.log_resource_state(
+                "collection",
+                collection_name,
+                "missing" if not upstream_exists else "exists",
+                "upstream",
+            )
+            assert not upstream_exists, (
+                f"Collection {collection_name} still exists in upstream after drop"
+            )
+
+            # Log sync verification start
+            self.log_sync_verification(
+                "DROP_COLLECTION", collection_name, "missing from downstream"
+            )
+
+            # Wait for drop to sync
+            def check_drop():
+                exists = downstream_client.has_collection(collection_name)
+                if not exists:
+                    self.log_resource_state(
+                        "collection",
+                        collection_name,
+                        "missing",
+                        "downstream",
+                        "Drop synced",
+                    )
+                return not exists
+
+            sync_success = self.wait_for_sync(
+                check_drop, sync_timeout, f"drop collection {collection_name}"
+            )
+            assert sync_success, (
+                f"Collection {collection_name} drop failed to sync to downstream"
+            )
+
+            # Log test success
+            duration = time.time() - start_time
+            self.log_test_end("test_drop_collection", True, duration)
+
+        except Exception as e:
+            duration = time.time() - start_time
+            logger.error(f"[ERROR] Test failed with error: {e}")
+            self.log_test_end("test_drop_collection", False, duration)
+            raise
+
+    def test_rename_collection(self, upstream_client, downstream_client, sync_timeout):
+        """Test RENAME_COLLECTION operation sync."""
+        start_time = time.time()
+
+        old_name = self.gen_unique_name("test_col_rename_old")
+        new_name = self.gen_unique_name("test_col_rename_new")
+
+        # Log test start
+        self.log_test_start(
+            "test_rename_collection", "RENAME_COLLECTION", f"{old_name} -> {new_name}"
+        )
+
+        # Store upstream client for teardown
+        self._upstream_client = upstream_client
+        self.resources_to_cleanup.append(("collection", old_name))
+        self.resources_to_cleanup.append(("collection", new_name))
+
+        try:
+            # Initial cleanup
+            self.cleanup_collection(upstream_client, old_name)
+            self.cleanup_collection(upstream_client, new_name)
+
+            # Create collection first
+            self.log_operation("CREATE_COLLECTION", "collection", old_name, "upstream")
+            upstream_client.create_collection(
+                collection_name=old_name,
+                schema=self.create_default_schema(upstream_client),
+            )
+
+            # Wait for creation to sync
+            def check_create():
+                return downstream_client.has_collection(old_name)
+
+            assert self.wait_for_sync(
+                check_create, sync_timeout, f"create collection {old_name}"
+            )
+
+            # Rename collection
+            rename_start_time = time.time()
+            self.log_operation(
+                "RENAME_COLLECTION",
+                "collection",
+                f"{old_name} -> {new_name}",
+                "upstream",
+            )
+
+            try:
+                upstream_client.rename_collection(old_name, new_name)
+                rename_duration = time.time() - rename_start_time
+                logger.info(
+                    f"[SUCCESS] Rename operation completed in {rename_duration:.2f}s"
+                )
+            except Exception as e:
+                rename_duration = time.time() - rename_start_time
+                logger.error(
+                    f"[FAILED] Rename operation failed after {rename_duration:.2f}s: {e}"
+                )
+                raise
+
+            # Verify rename in upstream
+            old_exists = upstream_client.has_collection(old_name)
+            new_exists = upstream_client.has_collection(new_name)
+            self.log_resource_state(
+                "collection",
+                old_name,
+                "missing" if not old_exists else "exists",
+                "upstream",
+            )
+            self.log_resource_state(
+                "collection",
+                new_name,
+                "exists" if new_exists else "missing",
+                "upstream",
+            )
+
+            assert not old_exists, (
+                f"Old collection {old_name} still exists after rename"
+            )
+            assert new_exists, f"New collection {new_name} not found after rename"
+
+            # Log sync verification start
+            self.log_sync_verification(
+                "RENAME_COLLECTION",
+                f"{old_name} -> {new_name}",
+                "completed in downstream",
+            )
+
+            # Wait for rename to sync
+            def check_rename():
+                return not downstream_client.has_collection(
+                    old_name
+                ) and downstream_client.has_collection(new_name)
+
+            sync_success = self.wait_for_sync(
+                check_rename,
+                sync_timeout,
+                f"rename collection {old_name} to {new_name}",
+            )
+            assert sync_success, (
+                f"Collection rename from {old_name} to {new_name} failed to sync to downstream"
+            )
+
+            # Log test success
+            duration = time.time() - start_time
+            self.log_test_end("test_rename_collection", True, duration)
+
+        except Exception as e:
+            duration = time.time() - start_time
+            logger.error(f"[ERROR] Test failed with error: {e}")
+            self.log_test_end("test_rename_collection", False, duration)
+            raise
+
+
+@pytest.mark.tags(CaseLabel.CDC)
+class TestCDCSyncCollectionManagement(TestCDCSyncBase):
+    """Test CDC sync for collection management operations."""
+
+    def setup_method(self):
+        """Setup for each test method."""
+        self.resources_to_cleanup = []
+
+    def teardown_method(self):
+        """Cleanup after each test method - only cleanup upstream, downstream will sync."""
+        upstream_client = getattr(self, "_upstream_client", None)
+
+        if upstream_client:
+            for resource_type, resource_name in self.resources_to_cleanup:
+                if resource_type == "collection":
+                    self.cleanup_collection(upstream_client, resource_name)
+
+            time.sleep(1)  # Allow cleanup to sync to downstream
+
+    def test_add_collection_field(
+        self, upstream_client, downstream_client, sync_timeout
+    ):
+        """Test ADD_FIELD operation sync."""
+        # Store upstream client for teardown
+        self._upstream_client = upstream_client
+
+        collection_name = self.gen_unique_name("test_col_add_field")
+        self.resources_to_cleanup.append(("collection", collection_name))
+
+        # Initial cleanup
+        self.cleanup_collection(upstream_client, collection_name)
+
+        # Create collection
+        schema = self.create_default_schema(upstream_client)
+        upstream_client.create_collection(
+            collection_name=collection_name, schema=schema, consistency_level="Strong"
+        )
+        assert self.wait_for_sync(
+            lambda: upstream_client.has_collection(collection_name),
+            sync_timeout,
+            f"create collection {collection_name}",
+        )
+
+        # Add field
+        upstream_client.add_collection_field(
+            collection_name,
+            field_name="new_field",
+            data_type=DataType.INT64,
+            nullable=True,
+        )
+        print(f"DEBUG: add field {collection_name}")
+        res = upstream_client.describe_collection(collection_name)
+        print(f"DEBUG: describe collection {collection_name}: {res}")
+
+        # Wait for addition to sync
+        def check_add():
+            res = downstream_client.describe_collection(collection_name)
+            logger.info(
+                f"DEBUG: describe collection in downstream {collection_name}: {res}"
+            )
+            return "new_field" in [field["name"] for field in res["fields"]]
+
+        assert self.wait_for_sync(
+            check_add, sync_timeout, f"add field {collection_name}"
+        )
+
+    def test_load_collection(self, upstream_client, downstream_client, sync_timeout):
+        """Test LOAD_COLLECTION operation sync."""
+        # Store upstream client for teardown
+        self._upstream_client = upstream_client
+
+        collection_name = self.gen_unique_name("test_col_load")
+        self.resources_to_cleanup.append(("collection", collection_name))
+
+        # Initial cleanup
+        self.cleanup_collection(upstream_client, collection_name)
+
+        # Create collection with proper schema
+        schema = self.create_default_schema(upstream_client)
+        upstream_client.create_collection(
+            collection_name=collection_name, schema=schema, consistency_level="Strong"
+        )
+
+        # Create index (required for loading)
+        index_params = upstream_client.prepare_index_params()
+        index_params.add_index(
+            field_name="vector", index_type="AUTOINDEX", metric_type="L2"
+        )
+        upstream_client.create_index(collection_name, index_params)
+
+        # Wait for creation to sync
+        def check_create():
+            return downstream_client.has_collection(collection_name)
+
+        assert self.wait_for_sync(
+            check_create, sync_timeout, f"create collection {collection_name}"
+        )
+
+        # Load collection
+        upstream_client.load_collection(collection_name)
+
+        # Wait for load to sync
+        def check_load():
+            try:
+                # Try to perform a search to verify the collection is loaded
+                query_vector = [[0.1] * 128]  # dummy vector
+                downstream_client.search(
+                    collection_name=collection_name,
+                    data=query_vector,
+                    limit=1,
+                    output_fields=[],
+                )
+                return True
+            except:
+                return False
+
+        assert self.wait_for_sync(
+            check_load, sync_timeout, f"load collection {collection_name}"
+        )
+
+    @pytest.mark.skip(reason="skip multi-replica test")
+    def test_load_collection_multi_replicas(
+        self, upstream_client, downstream_client, sync_timeout
+    ):
+        """Test LOAD_COLLECTION operation with multiple replicas sync."""
+        # Store upstream client for teardown
+        self._upstream_client = upstream_client
+
+        collection_name = self.gen_unique_name("test_col_multi_replicas")
+        self.resources_to_cleanup.append(("collection", collection_name))
+
+        # Initial cleanup
+        self.cleanup_collection(upstream_client, collection_name)
+
+        # Create collection with proper schema
+        schema = self.create_default_schema(upstream_client)
+        upstream_client.create_collection(
+            collection_name=collection_name, schema=schema, consistency_level="Strong"
+        )
+
+        # Create index (required for loading)
+        index_params = upstream_client.prepare_index_params()
+        index_params.add_index(
+            field_name="vector", index_type="AUTOINDEX", metric_type="L2"
+        )
+        upstream_client.create_index(collection_name, index_params)
+
+        # Wait for creation to sync
+        def check_create():
+            return downstream_client.has_collection(collection_name)
+
+        assert self.wait_for_sync(
+            check_create, sync_timeout, f"create collection {collection_name}"
+        )
+
+        # Load collection with 2 replicas
+        replica_number = 2
+        logger.info(
+            f"Loading collection {collection_name} with {replica_number} replicas"
+        )
+        upstream_client.load_collection(collection_name, replica_number=replica_number)
+
+        # Verify upstream load with replicas and check replica count
+        def verify_upstream_replicas():
+            try:
+                # Create Collection object to get replica information
+                upstream_collection = Collection(
+                    name=collection_name, using=upstream_client._using
+                )
+
+                # Get replicas information
+                replicas = upstream_collection.get_replicas()
+                actual_replica_count = len(replicas.groups)
+                logger.info(
+                    f"Upstream collection {collection_name} has {actual_replica_count} replicas"
+                )
+                logger.info(f"Replica details: {replicas}")
+
+                # Verify replica count matches expected
+                if actual_replica_count != replica_number:
+                    logger.warning(
+                        f"Expected {replica_number} replicas, but found {actual_replica_count}"
+                    )
+                    return False
+
+                # Try to perform a search to verify the collection is loaded
+                query_vector = [[0.1] * 128]
+                upstream_client.search(
+                    collection_name=collection_name,
+                    data=query_vector,
+                    limit=1,
+                    output_fields=[],
+                )
+                logger.info(
+                    f"Upstream collection {collection_name} loaded successfully with {actual_replica_count} replicas"
+                )
+                return True
+            except Exception as e:
+                logger.warning(f"Upstream load verification failed: {e}")
+                return False
+
+        assert self.wait_for_sync(
+            verify_upstream_replicas,
+            sync_timeout,
+            f"load collection {collection_name} with {replica_number} replicas in upstream",
+        )
+
+        # Wait for load with replicas to sync to downstream
+        def check_downstream_replicas():
+            try:
+                # Create Collection object to get replica information
+                downstream_collection = Collection(
+                    name=collection_name, using=downstream_client._using
+                )
+
+                # Get replicas information
+                replicas = downstream_collection.get_replicas()
+                actual_replica_count = len(replicas.groups)
+                logger.info(
+                    f"Downstream collection {collection_name} has {actual_replica_count} replicas"
+                )
+                logger.info(f"Replica details: {replicas}")
+
+                # Verify replica count matches expected
+                if actual_replica_count != replica_number:
+                    logger.warning(
+                        f"Expected {replica_number} replicas in downstream, but found {actual_replica_count}"
+                    )
+                    return False
+
+                # Try to perform a search to verify the collection is loaded in downstream
+                query_vector = [[0.1] * 128]
+                downstream_client.search(
+                    collection_name=collection_name,
+                    data=query_vector,
+                    limit=1,
+                    output_fields=[],
+                )
+                logger.info(
+                    f"Downstream collection {collection_name} loaded successfully with {actual_replica_count} replicas"
+                )
+                return True
+            except Exception as e:
+                logger.warning(f"Downstream load check failed: {e}")
+                return False
+
+        assert self.wait_for_sync(
+            check_downstream_replicas,
+            sync_timeout,
+            f"load collection {collection_name} with {replica_number} replicas in downstream",
+        )
+
+        logger.info(
+            f"Successfully verified multi-replica load sync for collection {collection_name}"
+        )
+        logger.info(f"Both upstream and downstream have {replica_number} replicas")
+
+        # Now test release with multiple replicas
+        logger.info(
+            f"Testing release operation for multi-replica collection {collection_name}"
+        )
+        upstream_client.release_collection(collection_name)
+
+        # Verify upstream release
+        def verify_upstream_release():
+            try:
+                # Try to search - should fail if released
+                query_vector = [[0.1] * 128]
+                upstream_client.search(
+                    collection_name=collection_name,
+                    data=query_vector,
+                    limit=1,
+                    output_fields=[],
+                )
+                logger.warning(
+                    f"Upstream collection {collection_name} is still loaded (search succeeded)"
+                )
+                return False  # If search succeeds, collection is still loaded
+            except Exception as e:
+                logger.info(
+                    f"Upstream collection {collection_name} released successfully (search failed as expected): {e}"
+                )
+                return True  # If search fails, collection is released
+
+        assert self.wait_for_sync(
+            verify_upstream_release,
+            sync_timeout,
+            f"release collection {collection_name} in upstream",
+        )
+
+        # Wait for release to sync to downstream
+        def check_downstream_release():
+            try:
+                # Try to search - should fail if released
+                query_vector = [[0.1] * 128]
+                downstream_client.search(
+                    collection_name=collection_name,
+                    data=query_vector,
+                    limit=1,
+                    output_fields=[],
+                )
+                logger.warning(
+                    f"Downstream collection {collection_name} is still loaded (search succeeded)"
+                )
+                return False  # If search succeeds, collection is still loaded
+            except Exception as e:
+                logger.info(
+                    f"Downstream collection {collection_name} released successfully (search failed as expected): {e}"
+                )
+                return True  # If search fails, collection is released
+
+        assert self.wait_for_sync(
+            check_downstream_release,
+            sync_timeout,
+            f"release collection {collection_name} in downstream",
+        )
+
+        logger.info(
+            f"Successfully verified multi-replica release sync for collection {collection_name}"
+        )
+        logger.info(
+            f"Both upstream and downstream released the collection with {replica_number} replicas"
+        )
+
+    def test_release_collection(self, upstream_client, downstream_client, sync_timeout):
+        """Test RELEASE_COLLECTION operation sync."""
+        # Store upstream client for teardown
+        self._upstream_client = upstream_client
+
+        collection_name = self.gen_unique_name("test_col_release")
+        self.resources_to_cleanup.append(("collection", collection_name))
+
+        # Initial cleanup
+        self.cleanup_collection(upstream_client, collection_name)
+
+        # Create collection with proper schema
+        schema = self.create_default_schema(upstream_client)
+        upstream_client.create_collection(
+            collection_name=collection_name, schema=schema, consistency_level="Strong"
+        )
+
+        index_params = upstream_client.prepare_index_params()
+        index_params.add_index(
+            field_name="vector", index_type="AUTOINDEX", metric_type="L2"
+        )
+        upstream_client.create_index(collection_name, index_params)
+        upstream_client.load_collection(collection_name)
+
+        # Wait for setup to sync
+        def check_setup():
+            try:
+                query_vector = [[0.1] * 128]
+                downstream_client.search(
+                    collection_name=collection_name,
+                    data=query_vector,
+                    limit=1,
+                    output_fields=[],
+                )
+                return True
+            except:
+                return False
+
+        assert self.wait_for_sync(
+            check_setup, sync_timeout, f"setup and load collection {collection_name}"
+        )
+
+        # Release collection
+        upstream_client.release_collection(collection_name)
+
+        # Wait for release to sync
+        def check_release():
+            try:
+                # Try to search - should fail if released
+                query_vector = [[0.1] * 128]
+                downstream_client.search(
+                    collection_name=collection_name,
+                    data=query_vector,
+                    limit=1,
+                    output_fields=[],
+                )
+                return False  # If search succeeds, collection is still loaded
+            except:
+                return True  # If search fails, collection is released
+
+        assert self.wait_for_sync(
+            check_release, sync_timeout, f"release collection {collection_name}"
+        )
+
+    def test_flush(self, upstream_client, downstream_client, sync_timeout):
+        """Test FLUSH operation sync."""
+        # Store upstream client for teardown
+        self._upstream_client = upstream_client
+
+        collection_name = self.gen_unique_name("test_col_flush")
+        self.resources_to_cleanup.append(("collection", collection_name))
+
+        # Initial cleanup
+        self.cleanup_collection(upstream_client, collection_name)
+
+        # Create collection with proper schema
+        schema = self.create_default_schema(upstream_client)
+        upstream_client.create_collection(
+            collection_name=collection_name, schema=schema, consistency_level="Strong"
+        )
+
+        # Wait for creation to sync
+        def check_create():
+            return downstream_client.has_collection(collection_name)
+
+        assert self.wait_for_sync(
+            check_create, sync_timeout, f"create collection {collection_name}"
+        )
+
+        # Insert data (without immediate flush)
+        test_data = self.generate_test_data(5000)
+        insert_result = upstream_client.insert(collection_name, test_data)
+
+        # Verify data is not visible before flush
+        stats_before = upstream_client.get_collection_stats(collection_name)
+        logger.info(f"Stats before flush: {stats_before}")
+
+        # Flush collection
+        upstream_client.flush(collection_name)
+
+        # Wait for flush data to be visible with timeout
+        expected_count = (
+            insert_result.get("insert_count", 100) if insert_result else 100
+        )
+
+        def check_flush_stats():
+            try:
+                stats = upstream_client.get_collection_stats(collection_name)
+                logger.info(
+                    f"DEBUG: get collection stats in upstream {collection_name}: {stats}"
+                )
+                row_count = stats.get("row_count", 0)
+                logger.info(
+                    f"Current row count: {row_count}, expected: {expected_count}"
+                )
+                return row_count >= expected_count
+            except Exception as e:
+                logger.warning(f"Error checking stats: {e}")
+                return False
+
+        # Use timeout for waiting flush stats to update
+        timeout = 30  # 30 seconds timeout
+        assert self.wait_for_sync(
+            check_flush_stats,
+            timeout,
+            f"flush data visible in stats (expected: {expected_count})",
+        )
+
+        # Get final stats after flush
+        stats_after = upstream_client.get_collection_stats(collection_name)
+        logger.info(f"Stats after flush: {stats_after}")
+
+        # Wait for flush to sync downstream
+        def check_flush():
+            try:
+                downstream_stats = downstream_client.get_collection_stats(
+                    collection_name
+                )
+                logger.info(
+                    f"DEBUG: get collection stats in downstream {collection_name}: {downstream_stats}"
+                )
+                return downstream_stats.get("row_count", 0) >= 100
+            except:
+                return False
+
+        assert self.wait_for_sync(
+            check_flush, sync_timeout, f"flush collection {collection_name}"
+        )
+
+    def test_load_collection_with_load_fields(
+        self, upstream_client, downstream_client, sync_timeout
+    ):
+        """Test LOAD_COLLECTION operation with load_fields parameter sync."""
+        # Store upstream client for teardown
+        self._upstream_client = upstream_client
+
+        collection_name = self.gen_unique_name("test_col_load_fields")
+        self.resources_to_cleanup.append(("collection", collection_name))
+
+        # Initial cleanup
+        self.cleanup_collection(upstream_client, collection_name)
+
+        # Create collection with comprehensive schema (has multiple fields)
+        schema = self.create_comprehensive_schema(upstream_client)
+        upstream_client.create_collection(
+            collection_name=collection_name, schema=schema, consistency_level="Strong"
+        )
+
+        # Create index for float_vector field (required for loading)
+        index_params = upstream_client.prepare_index_params()
+        index_params.add_index(
+            field_name="float_vector", index_type="AUTOINDEX", metric_type="L2"
+        )
+        upstream_client.create_index(collection_name, index_params)
+
+        # Wait for creation to sync
+        def check_create():
+            return downstream_client.has_collection(collection_name)
+
+        assert self.wait_for_sync(
+            check_create, sync_timeout, f"create collection {collection_name}"
+        )
+
+        # Insert some test data
+        test_data = self.generate_comprehensive_test_data(100)
+        upstream_client.insert(collection_name, test_data)
+        upstream_client.flush(collection_name)
+
+        # Load collection with specific fields only (float_vector + id + varchar_field)
+        load_fields = ["float_vector", "id", "varchar_field"]
+        upstream_client.load_collection(collection_name, load_fields=load_fields)
+
+        # Verify upstream load operation succeeded
+        def verify_upstream_load():
+            try:
+                # Try to search to verify collection is loaded
+                query_vector = [[0.1] * 128]
+                upstream_client.search(
+                    collection_name=collection_name,
+                    data=query_vector,
+                    limit=1,
+                    output_fields=["varchar_field"],
+                    anns_field="float_vector",
+                )
+                return True
+            except Exception as e:
+                logger.warning(f"Upstream load verification failed: {e}")
+                return False
+
+        assert self.wait_for_sync(
+            verify_upstream_load,
+            sync_timeout,
+            f"verify upstream load with load_fields in {collection_name}",
+        )
+
+        # Verify downstream sync - collection should be loaded and searchable
+        def check_downstream_load_sync():
+            try:
+                query_vector = [[0.1] * 128]
+                result = downstream_client.search(
+                    collection_name=collection_name,
+                    data=query_vector,
+                    limit=1,
+                    output_fields=["varchar_field"],
+                    anns_field="float_vector",
+                )
+                return len(result) > 0 and len(result[0]) >= 0
+            except Exception as e:
+                logger.warning(f"Downstream load sync check failed: {e}")
+                return False
+
+        assert self.wait_for_sync(
+            check_downstream_load_sync,
+            sync_timeout,
+            f"verify downstream load sync for {collection_name}",
+        )
+
+        # Additional verification: test that both loaded and unloaded fields can be output
+        # (since load_fields only affects memory usage, not field accessibility)
+        def verify_all_fields_accessible():
+            try:
+                query_vector = [[0.1] * 128]
+                # Test accessing both loaded and unloaded fields
+                result1 = downstream_client.search(
+                    collection_name=collection_name,
+                    data=query_vector,
+                    limit=1,
+                    output_fields=["varchar_field"],  # loaded field
+                    anns_field="float_vector",
+                )
+
+                result2 = downstream_client.search(
+                    collection_name=collection_name,
+                    data=query_vector,
+                    limit=1,
+                    output_fields=[
+                        "float_field"
+                    ],  # unloaded field (but should still be accessible)
+                    anns_field="float_vector",
+                )
+
+                return len(result1) > 0 and len(result2) > 0
+            except Exception as e:
+                logger.warning(f"Field accessibility verification failed: {e}")
+                return False
+
+        assert self.wait_for_sync(
+            verify_all_fields_accessible,
+            sync_timeout,
+            f"verify all fields accessible in {collection_name}",
+        )
+
+        logger.info(
+            f"Successfully tested load_collection with load_fields: {load_fields}"
+        )
+        logger.info("Verified CDC sync of load operation with load_fields parameter")

tests/python_client/cdc/testcases/test_collection_properties.py

@@ -0,0 +1,276 @@
+"""
+CDC sync tests for collection property operations.
+"""
+
+import time
+
+from .base import TestCDCSyncBase, logger
+
+
+class TestCDCSyncCollectionProperties(TestCDCSyncBase):
+    """Test CDC sync for collection property (alter/drop) operations."""
+
+    def setup_method(self):
+        """Setup for each test method."""
+        self.resources_to_cleanup = []
+
+    def teardown_method(self):
+        """Cleanup after each test method - only cleanup upstream, downstream will sync."""
+        upstream_client = getattr(self, "_upstream_client", None)
+
+        if upstream_client:
+            for resource_type, resource_name in self.resources_to_cleanup:
+                if resource_type == "collection":
+                    self.cleanup_collection(upstream_client, resource_name)
+
+            time.sleep(1)  # Allow cleanup to sync to downstream
+
+    def _create_basic_collection(self, client, c_name):
+        """Create a default schema collection, insert 100 rows, and create HNSW index."""
+        schema = self.create_default_schema(client)
+        client.create_collection(
+            collection_name=c_name,
+            schema=schema,
+            consistency_level="Strong",
+        )
+
+        # Insert 100 rows
+        test_data = self.generate_test_data(100)
+        client.insert(c_name, test_data)
+
+        # Create HNSW index on vector field
+        index_params = client.prepare_index_params()
+        index_params.add_index(
+            field_name="vector",
+            index_type="HNSW",
+            metric_type="L2",
+            params={"M": 8, "efConstruction": 64},
+        )
+        client.create_index(c_name, index_params)
+
+    def test_ttl_sync(self, upstream_client, downstream_client, sync_timeout):
+        """Test ALTER_COLLECTION_PROPERTIES (TTL) sync."""
+        # Store upstream client for teardown
+        self._upstream_client = upstream_client
+
+        c_name = self.gen_unique_name("test_col_ttl")
+        self.resources_to_cleanup.append(("collection", c_name))
+
+        # Initial cleanup
+        self.cleanup_collection(upstream_client, c_name)
+
+        # Create collection
+        self._create_basic_collection(upstream_client, c_name)
+
+        # Wait for collection to sync downstream
+        def check_create():
+            return downstream_client.has_collection(c_name)
+
+        assert self.wait_for_sync(check_create, sync_timeout, f"create collection {c_name}")
+
+        # Alter collection TTL property
+        upstream_client.alter_collection_properties(
+            collection_name=c_name,
+            properties={"collection.ttl.seconds": "3600"},
+        )
+
+        # Wait for property to sync downstream
+        def check_ttl():
+            try:
+                desc = downstream_client.describe_collection(c_name)
+                props = desc.get("properties", {})
+                logger.info(f"Downstream collection properties: {props}")
+                return str(props.get("collection.ttl.seconds", "")) == "3600"
+            except Exception as e:
+                logger.warning(f"Check TTL sync failed: {e}")
+                return False
+
+        assert self.wait_for_sync(check_ttl, sync_timeout, f"TTL property sync for {c_name}")
+
+    def test_mmap_sync(self, upstream_client, downstream_client, sync_timeout):
+        """Test ALTER_COLLECTION_PROPERTIES (mmap.enabled) sync."""
+        # Store upstream client for teardown
+        self._upstream_client = upstream_client
+
+        c_name = self.gen_unique_name("test_col_mmap")
+        self.resources_to_cleanup.append(("collection", c_name))
+
+        # Initial cleanup
+        self.cleanup_collection(upstream_client, c_name)
+
+        # Create collection
+        self._create_basic_collection(upstream_client, c_name)
+
+        # Wait for collection to sync downstream
+        def check_create():
+            return downstream_client.has_collection(c_name)
+
+        assert self.wait_for_sync(check_create, sync_timeout, f"create collection {c_name}")
+
+        # Alter mmap.enabled property
+        upstream_client.alter_collection_properties(
+            collection_name=c_name,
+            properties={"mmap.enabled": "true"},
+        )
+
+        # Wait for property to sync downstream
+        def check_mmap():
+            try:
+                desc = downstream_client.describe_collection(c_name)
+                props = desc.get("properties", {})
+                logger.info(f"Downstream collection properties: {props}")
+                return str(props.get("mmap.enabled", "")).lower() == "true"
+            except Exception as e:
+                logger.warning(f"Check mmap sync failed: {e}")
+                return False
+
+        assert self.wait_for_sync(check_mmap, sync_timeout, f"mmap property sync for {c_name}")
+
+    def test_autocompaction_sync(self, upstream_client, downstream_client, sync_timeout):
+        """Test ALTER_COLLECTION_PROPERTIES (autocompaction.enabled) sync."""
+        # Store upstream client for teardown
+        self._upstream_client = upstream_client
+
+        c_name = self.gen_unique_name("test_col_autocomp")
+        self.resources_to_cleanup.append(("collection", c_name))
+
+        # Initial cleanup
+        self.cleanup_collection(upstream_client, c_name)
+
+        # Create collection
+        self._create_basic_collection(upstream_client, c_name)
+
+        # Wait for collection to sync downstream
+        def check_create():
+            return downstream_client.has_collection(c_name)
+
+        assert self.wait_for_sync(check_create, sync_timeout, f"create collection {c_name}")
+
+        # Alter autocompaction property
+        upstream_client.alter_collection_properties(
+            collection_name=c_name,
+            properties={"collection.autocompaction.enabled": "true"},
+        )
+
+        # Wait for property to sync downstream
+        def check_autocomp():
+            try:
+                desc = downstream_client.describe_collection(c_name)
+                props = desc.get("properties", {})
+                logger.info(f"Downstream collection properties: {props}")
+                return str(props.get("collection.autocompaction.enabled", "")).lower() == "true"
+            except Exception as e:
+                logger.warning(f"Check autocompaction sync failed: {e}")
+                return False
+
+        assert self.wait_for_sync(check_autocomp, sync_timeout, f"autocompaction property sync for {c_name}")
+
+    def test_alter_multiple_properties(self, upstream_client, downstream_client, sync_timeout):
+        """Test ALTER_COLLECTION_PROPERTIES with multiple properties at once sync."""
+        # Store upstream client for teardown
+        self._upstream_client = upstream_client
+
+        c_name = self.gen_unique_name("test_col_multi_props")
+        self.resources_to_cleanup.append(("collection", c_name))
+
+        # Initial cleanup
+        self.cleanup_collection(upstream_client, c_name)
+
+        # Create collection
+        self._create_basic_collection(upstream_client, c_name)
+
+        # Wait for collection to sync downstream
+        def check_create():
+            return downstream_client.has_collection(c_name)
+
+        assert self.wait_for_sync(check_create, sync_timeout, f"create collection {c_name}")
+
+        # Alter all 3 properties at once
+        upstream_client.alter_collection_properties(
+            collection_name=c_name,
+            properties={
+                "collection.ttl.seconds": "3600",
+                "mmap.enabled": "true",
+                "collection.autocompaction.enabled": "true",
+            },
+        )
+
+        # Wait for all 3 properties to sync downstream
+        def check_all_props():
+            try:
+                desc = downstream_client.describe_collection(c_name)
+                props = desc.get("properties", {})
+                logger.info(f"Downstream collection properties: {props}")
+                ttl_ok = str(props.get("collection.ttl.seconds", "")) == "3600"
+                mmap_ok = str(props.get("mmap.enabled", "")).lower() == "true"
+                autocomp_ok = str(props.get("collection.autocompaction.enabled", "")).lower() == "true"
+                return ttl_ok and mmap_ok and autocomp_ok
+            except Exception as e:
+                logger.warning(f"Check all properties sync failed: {e}")
+                return False
+
+        assert self.wait_for_sync(check_all_props, sync_timeout, f"all properties sync for {c_name}")
+
+    def test_drop_properties_sync(self, upstream_client, downstream_client, sync_timeout):
+        """Test DROP_COLLECTION_PROPERTIES — set TTL + mmap, drop TTL, verify TTL gone and mmap remains."""
+        # Store upstream client for teardown
+        self._upstream_client = upstream_client
+
+        c_name = self.gen_unique_name("test_col_drop_props")
+        self.resources_to_cleanup.append(("collection", c_name))
+
+        # Initial cleanup
+        self.cleanup_collection(upstream_client, c_name)
+
+        # Create collection
+        self._create_basic_collection(upstream_client, c_name)
+
+        # Wait for collection to sync downstream
+        def check_create():
+            return downstream_client.has_collection(c_name)
+
+        assert self.wait_for_sync(check_create, sync_timeout, f"create collection {c_name}")
+
+        # Set TTL and mmap properties
+        upstream_client.alter_collection_properties(
+            collection_name=c_name,
+            properties={
+                "collection.ttl.seconds": "3600",
+                "mmap.enabled": "true",
+            },
+        )
+
+        # Wait for both properties to sync downstream
+        def check_props_set():
+            try:
+                desc = downstream_client.describe_collection(c_name)
+                props = desc.get("properties", {})
+                ttl_set = str(props.get("collection.ttl.seconds", "")) == "3600"
+                mmap_set = str(props.get("mmap.enabled", "")).lower() == "true"
+                return ttl_set and mmap_set
+            except Exception as e:
+                logger.warning(f"Check properties set failed: {e}")
+                return False
+
+        assert self.wait_for_sync(check_props_set, sync_timeout, f"set properties for {c_name}")
+
+        # Drop TTL property
+        upstream_client.drop_collection_properties(
+            collection_name=c_name,
+            property_keys=["collection.ttl.seconds"],
+        )
+
+        # Wait for TTL to be gone but mmap to remain on downstream
+        def check_drop_ttl():
+            try:
+                desc = downstream_client.describe_collection(c_name)
+                props = desc.get("properties", {})
+                logger.info(f"Downstream collection properties after drop: {props}")
+                ttl_gone = "collection.ttl.seconds" not in props
+                mmap_remains = str(props.get("mmap.enabled", "")).lower() == "true"
+                return ttl_gone and mmap_remains
+            except Exception as e:
+                logger.warning(f"Check drop TTL sync failed: {e}")
+                return False
+
+        assert self.wait_for_sync(check_drop_ttl, sync_timeout, f"drop TTL property sync for {c_name}")

tests/python_client/cdc/testcases/test_database.py

@@ -0,0 +1,256 @@
+"""
+CDC sync tests for database operations.
+"""
+
+import time
+import pytest
+from common.common_type import CaseLabel
+from .base import TestCDCSyncBase, logger
+
+
+@pytest.mark.tags(CaseLabel.CDC)
+class TestCDCSyncDatabase(TestCDCSyncBase):
+    """Test CDC sync for database operations."""
+
+    def setup_method(self):
+        """Setup for each test method."""
+        self.resources_to_cleanup = []
+
+    def teardown_method(self):
+        """Cleanup after each test method - only cleanup upstream, downstream will sync."""
+        upstream_client = getattr(self, "_upstream_client", None)
+
+        if upstream_client:
+            for resource_type, resource_name in self.resources_to_cleanup:
+                if resource_type == "database":
+                    self.cleanup_database(upstream_client, resource_name)
+
+            time.sleep(1)  # Allow cleanup to sync to downstream
+
+    def test_create_database(self, upstream_client, downstream_client, sync_timeout):
+        """Test CREATE_DATABASE operation sync."""
+        start_time = time.time()
+        db_name = self.gen_unique_name("test_db_create")
+
+        # Log test start
+        self.log_test_start("test_create_database", "CREATE_DATABASE", db_name)
+
+        # Store upstream client for teardown
+        self._upstream_client = upstream_client
+        self.resources_to_cleanup.append(("database", db_name))
+
+        try:
+            # Initial cleanup
+            self.cleanup_database(upstream_client, db_name)
+
+            # Log operation
+            self.log_operation("CREATE_DATABASE", "database", db_name, "upstream")
+
+            # Create database in upstream
+            upstream_client.create_database(db_name)
+
+            # Verify upstream creation
+            upstream_databases = upstream_client.list_databases()
+            upstream_exists = db_name in upstream_databases
+            self.log_resource_state(
+                "database",
+                db_name,
+                "exists" if upstream_exists else "missing",
+                "upstream",
+            )
+            assert upstream_exists, f"Database {db_name} not created in upstream"
+
+            # Log sync verification start
+            self.log_sync_verification(
+                "CREATE_DATABASE", db_name, "exists in downstream"
+            )
+
+            # Wait for sync to downstream
+            def check_sync():
+                downstream_databases = downstream_client.list_databases()
+                exists = db_name in downstream_databases
+                if exists:
+                    self.log_resource_state(
+                        "database", db_name, "exists", "downstream", "Sync confirmed"
+                    )
+                return exists
+
+            sync_success = self.wait_for_sync(
+                check_sync, sync_timeout, f"create database {db_name}"
+            )
+            assert sync_success, f"Database {db_name} failed to sync to downstream"
+
+            # Log test success
+            duration = time.time() - start_time
+            self.log_test_end("test_create_database", True, duration)
+
+        except Exception as e:
+            duration = time.time() - start_time
+            logger.error(f"[ERROR] Test failed with error: {e}")
+            self.log_test_end("test_create_database", False, duration)
+            raise
+
+    def test_drop_database(self, upstream_client, downstream_client, sync_timeout):
+        """Test DROP_DATABASE operation sync."""
+        # Store upstream client for teardown
+        self._upstream_client = upstream_client
+
+        db_name = self.gen_unique_name("test_db_drop")
+        self.resources_to_cleanup.append(("database", db_name))
+
+        # Initial cleanup
+        self.cleanup_database(upstream_client, db_name)
+
+        # Create database in upstream first
+        upstream_client.create_database(db_name)
+
+        # Wait for creation to sync
+        def check_create():
+            return db_name in downstream_client.list_databases()
+
+        assert self.wait_for_sync(
+            check_create, sync_timeout, f"create database {db_name}"
+        )
+
+        # Drop database in upstream
+        upstream_client.drop_database(db_name)
+        assert db_name not in upstream_client.list_databases()
+
+        # Wait for drop to sync to downstream
+        def check_drop():
+            return db_name not in downstream_client.list_databases()
+
+        assert self.wait_for_sync(check_drop, sync_timeout, f"drop database {db_name}")
+
+    def test_alter_database_properties(
+        self, upstream_client, downstream_client, sync_timeout
+    ):
+        """Test ALTER_DATABASE_PROPERTIES operation sync."""
+        # Store upstream client for teardown
+        self._upstream_client = upstream_client
+
+        db_name = self.gen_unique_name("test_db_alter_props")
+        self.resources_to_cleanup.append(("database", db_name))
+
+        # Initial cleanup
+        self.cleanup_database(upstream_client, db_name)
+
+        # Create database in upstream first
+        upstream_client.create_database(db_name)
+
+        # Wait for creation to sync
+        def check_create():
+            return db_name in downstream_client.list_databases()
+
+        assert self.wait_for_sync(
+            check_create, sync_timeout, f"create database {db_name}"
+        )
+
+        # Set multiple database properties
+        properties_to_set = {
+            "database.max.collections": 100,
+            "database.diskQuota.mb": 1024,
+        }
+
+        upstream_client.alter_database_properties(
+            db_name=db_name, properties=properties_to_set
+        )
+
+        # Wait for alter properties to sync
+        def check_alter_properties():
+            try:
+                # Verify database exists
+                if db_name not in downstream_client.list_databases():
+                    return False
+
+                # Verify properties are synced correctly
+                downstream_props = downstream_client.describe_database(db_name)
+                logger.info(f"Downstream database properties: {downstream_props}")
+                for key, expected_value in properties_to_set.items():
+                    if key not in downstream_props or str(downstream_props[key]) != str(
+                        expected_value
+                    ):
+                        return False
+                return True
+            except:
+                return False
+
+        assert self.wait_for_sync(
+            check_alter_properties, sync_timeout, f"alter database properties {db_name}"
+        )
+        logger.info(f"Database properties altered successfully for {db_name}")
+
+    def test_drop_database_properties(
+        self, upstream_client, downstream_client, sync_timeout
+    ):
+        """Test DROP_DATABASE_PROPERTIES operation sync."""
+        # Store upstream client for teardown
+        self._upstream_client = upstream_client
+
+        db_name = self.gen_unique_name("test_db_drop_props")
+        self.resources_to_cleanup.append(("database", db_name))
+
+        # Initial cleanup
+        self.cleanup_database(upstream_client, db_name)
+
+        # Create database in upstream first
+        upstream_client.create_database(db_name)
+
+        # Wait for creation to sync
+        def check_create():
+            return db_name in downstream_client.list_databases()
+
+        assert self.wait_for_sync(
+            check_create, sync_timeout, f"create database {db_name}"
+        )
+
+        # First set some properties
+        properties_to_set = {
+            "database.max.collections": 200,
+            "database.diskQuota.mb": 2048,
+        }
+
+        upstream_client.alter_database_properties(
+            db_name=db_name, properties=properties_to_set
+        )
+
+        # Wait for initial properties to sync
+        time.sleep(3)  # Allow properties to be set
+
+        # Drop specific database properties (only drop one, keep the other)
+        property_keys_to_drop = ["database.max.collections"]
+
+        upstream_client.drop_database_properties(
+            db_name=db_name, property_keys=property_keys_to_drop
+        )
+
+        # Wait for drop properties to sync
+        def check_drop_properties():
+            try:
+                # Verify database exists
+                if db_name not in downstream_client.list_databases():
+                    return False
+
+                # Verify properties are synced correctly after drop
+                downstream_props = downstream_client.describe_database(db_name)
+
+                logger.info(f"Downstream database properties: {downstream_props}")
+                # Verify dropped property is not present
+                if "database.max.collections" in downstream_props:
+                    return False
+
+                # Verify non-dropped property is still present with correct value
+                if (
+                    "database.diskQuota.mb" not in downstream_props
+                    or str(downstream_props["database.diskQuota.mb"]) != "2048"
+                ):
+                    return False
+
+                return True
+            except:
+                return False
+
+        assert self.wait_for_sync(
+            check_drop_properties, sync_timeout, f"drop database properties {db_name}"
+        )
+        logger.info(f"Database properties dropped successfully for {db_name}")

tests/python_client/cdc/testcases/test_force_promote.py

@@ -0,0 +1,984 @@
+"""
+CDC force-promote failover scenario tests.
+
+Each test exercises a path through the streamingnode replicate interceptor's
+SwitchReplicateMode + force-promote ack-callback flow:
+- basic           — kill source, promote secondary, verify writable
+- target_restart  — restart secondary mid-promote
+- source_restart  — restart primary mid-promote
+- incomplete_ddl  — release on primary then immediately promote (Ignore=true path)
+- network_partition — partition source↔target, then promote
+- endurance (skipif) — repeated promote/restore loop for ENDURANCE_DURATION_MINUTES
+
+Every test's `finally` restores the A→B topology via switchover_helper so
+subsequent tests start from a known state.
+"""
+
+import os
+import subprocess
+import tempfile
+import threading
+import time
+
+import pytest
+import yaml
+from common.common_type import CaseLabel
+
+from .base import TestCDCSyncBase, logger
+
+FAILOVER_TIMEOUT = int(os.getenv("FAILOVER_TIMEOUT", "180"))
+FORCE_PROMOTE_RPC_TIMEOUT = int(os.getenv("FORCE_PROMOTE_RPC_TIMEOUT", "60"))
+PROMOTE_RETRY_INTERVAL = 5
+DEFAULT_INSERT_COUNT = 200
+
+
+@pytest.mark.tags(CaseLabel.CDC)
+class TestCDCForcePromote(TestCDCSyncBase):
+    """Force-promote failover scenarios."""
+
+    def setup_method(self):
+        self.resources_to_cleanup = []
+        self._upstream_client = None
+        self._downstream_client = None
+
+    def teardown_method(self):
+        """Safety-net cleanup: iterates resources_to_cleanup if a test raised
+        before its own finally could run. Normal cleanup happens in each test's
+        finally block."""
+        for resource_type, resource_name in getattr(self, "resources_to_cleanup", []):
+            if resource_type != "collection":
+                continue
+            for client in (
+                getattr(self, "_upstream_client", None),
+                getattr(self, "_downstream_client", None),
+            ):
+                if client is None:
+                    continue
+                try:
+                    self.cleanup_collection(client, resource_name)
+                except Exception:
+                    pass
+
+    # ------------------------------------------------------------------
+    # Helpers
+    # ------------------------------------------------------------------
+
+    def do_force_promote(
+        self,
+        downstream_client,
+        target_cluster_id,
+        downstream_uri,
+        downstream_token,
+        pchannel_num,
+        timeout=FORCE_PROMOTE_RPC_TIMEOUT,
+    ):
+        """Call force_promote on downstream with a standalone-primary config."""
+        config = {
+            "clusters": [
+                {
+                    "cluster_id": target_cluster_id,
+                    "connection_param": {"uri": downstream_uri, "token": downstream_token},
+                    "pchannels": [f"{target_cluster_id}-rootcoord-dml_{i}" for i in range(pchannel_num)],
+                }
+            ],
+            "cross_cluster_topology": [],
+        }
+        downstream_client.update_replicate_configuration(
+            timeout=timeout,
+            force_promote=True,
+            **config,
+        )
+        logger.info(f"[FORCE_PROMOTE] called on {target_cluster_id} (standalone primary config)")
+
+    def is_already_primary_error(self, err):
+        return "force promote can only be used on secondary clusters" in str(
+            err
+        ) and "current cluster is primary" in str(err)
+
+    def promote_call_until_writable(
+        self,
+        downstream_client,
+        target_cluster_id,
+        downstream_uri,
+        downstream_token,
+        pchannel_num,
+        timeout=FAILOVER_TIMEOUT,
+    ):
+        """Retry force_promote + probe write until success or timeout.
+
+        A probe write creates+drops a throwaway collection. This proves
+        downstream is in standalone-primary mode and accepts DDL.
+        """
+        deadline = time.time() + timeout
+        last_err = None
+        while time.time() < deadline:
+            try:
+                try:
+                    remaining = max(1, int(deadline - time.time()))
+                    self.do_force_promote(
+                        downstream_client,
+                        target_cluster_id,
+                        downstream_uri,
+                        downstream_token,
+                        pchannel_num,
+                        timeout=min(FORCE_PROMOTE_RPC_TIMEOUT, remaining),
+                    )
+                except Exception as e:
+                    if not self.is_already_primary_error(e):
+                        raise
+                    logger.info("[FORCE_PROMOTE] target is already primary; probing writable state")
+                probe = self.gen_unique_name("promote_probe", max_length=40)
+                schema = self.create_default_schema(downstream_client)
+                downstream_client.create_collection(probe, schema=schema)
+                downstream_client.drop_collection(probe)
+                logger.info("[FORCE_PROMOTE] probe write succeeded — cluster writable")
+                return
+            except Exception as e:
+                last_err = e
+                logger.warning(f"[FORCE_PROMOTE_RETRY] failed: {e}")
+                sleep_seconds = min(PROMOTE_RETRY_INTERVAL, max(0, deadline - time.time()))
+                if sleep_seconds:
+                    time.sleep(sleep_seconds)
+        raise TimeoutError(f"force_promote did not become writable within {timeout}s; last error: {last_err}")
+
+    def cleanup_resources(self):
+        """Drop any collections registered in resources_to_cleanup."""
+        for resource_type, resource_name in self.resources_to_cleanup:
+            if resource_type != "collection":
+                continue
+            for client_label, client in (
+                ("upstream", self._upstream_client),
+                ("downstream", self._downstream_client),
+            ):
+                if client is None:
+                    continue
+                try:
+                    self.cleanup_collection(client, resource_name)
+                    logger.info(f"[CLEANUP] dropped {resource_name} on {client_label}")
+                except Exception as e:
+                    logger.warning(f"[CLEANUP] failed to drop {resource_name} on {client_label}: {e}")
+        self.resources_to_cleanup = []
+
+    def cleanup_downstream_only_collection(self, collection_name):
+        """Drop a collection created while downstream is force-promoted primary."""
+        client = self._downstream_client
+        if client is None:
+            return
+        try:
+            if client.has_collection(collection_name):
+                logger.info(f"[CLEANUP] Cleaning up downstream-only collection: {collection_name}")
+                client.drop_collection(collection_name)
+                logger.info(f"[SUCCESS] Downstream-only collection {collection_name} cleaned up successfully")
+        except Exception as e:
+            logger.warning(f"[FAILED] Failed to cleanup downstream-only collection {collection_name}: {e}")
+        finally:
+            self.resources_to_cleanup = [
+                resource for resource in self.resources_to_cleanup if resource != ("collection", collection_name)
+            ]
+
+    # ------------------------------------------------------------------
+    # Tests
+    # ------------------------------------------------------------------
+
+    def test_force_promote_basic(
+        self,
+        upstream_client,
+        downstream_client,
+        sync_timeout,
+        switchover_helper,
+        kubectl_helper,
+        source_cluster_id,
+        target_cluster_id,
+        upstream_uri,
+        upstream_token,
+        downstream_uri,
+        downstream_token,
+        pchannel_num,
+    ):
+        """Insert N → kill source → force_promote(downstream) → verify writable."""
+        start_time = time.time()
+        c_name = self.gen_unique_name("test_fp_basic", max_length=50)
+        c_after = f"{c_name}_after"
+
+        self.log_test_start("test_force_promote_basic", "FORCE_PROMOTE_BASIC", c_name)
+        self._upstream_client = upstream_client
+        self._downstream_client = downstream_client
+        self.resources_to_cleanup.append(("collection", c_name))
+        self.resources_to_cleanup.append(("collection", c_after))
+
+        try:
+            self.cleanup_collection(upstream_client, c_name)
+            schema = self.create_manual_id_schema(upstream_client)
+            upstream_client.create_collection(collection_name=c_name, schema=schema)
+            index_params = upstream_client.prepare_index_params()
+            index_params.add_index(
+                field_name="vector",
+                index_type="HNSW",
+                metric_type="L2",
+                params={"M": 16, "efConstruction": 200},
+            )
+            upstream_client.create_index(c_name, index_params)
+            upstream_client.load_collection(c_name)
+
+            assert self.wait_for_sync(
+                lambda: downstream_client.has_collection(c_name),
+                sync_timeout,
+                f"create collection {c_name}",
+            )
+
+            data = self.generate_test_data_with_id(DEFAULT_INSERT_COUNT, start_id=0)
+            upstream_client.insert(c_name, data)
+            upstream_client.flush(c_name)
+
+            def check_synced():
+                try:
+                    res = downstream_client.query(collection_name=c_name, filter="", output_fields=["count(*)"])
+                    cnt = res[0]["count(*)"] if res else 0
+                    logger.info(f"[SYNC_PROGRESS] downstream count: {cnt}/{DEFAULT_INSERT_COUNT}")
+                    return cnt >= DEFAULT_INSERT_COUNT
+                except Exception as e:
+                    logger.warning(f"sync check failed: {e}")
+                    return False
+
+            assert self.wait_for_sync(check_synced, sync_timeout, f"initial sync {c_name}")
+
+            # Kill source
+            logger.info(f"[FAILOVER] Killing source pods (instance={source_cluster_id})...")
+            kubectl_helper.delete_pods(source_cluster_id)
+
+            # Promote downstream — retry until writable
+            self.promote_call_until_writable(
+                downstream_client,
+                target_cluster_id,
+                downstream_uri,
+                downstream_token,
+                pchannel_num,
+            )
+
+            # On downstream (now standalone primary), create a new collection + insert
+            schema_after = self.create_manual_id_schema(downstream_client)
+            downstream_client.create_collection(collection_name=c_after, schema=schema_after)
+            idx_after = downstream_client.prepare_index_params()
+            idx_after.add_index(
+                field_name="vector",
+                index_type="HNSW",
+                metric_type="L2",
+                params={"M": 16, "efConstruction": 200},
+            )
+            downstream_client.create_index(c_after, idx_after)
+            downstream_client.load_collection(c_after)
+            data_after = self.generate_test_data_with_id(100, start_id=0)
+            downstream_client.insert(c_after, data_after)
+            downstream_client.flush(c_after)
+
+            res = downstream_client.query(collection_name=c_after, filter="", output_fields=["count(*)"])
+            assert res and res[0]["count(*)"] >= 100, f"downstream not writable after force_promote: count={res}"
+        finally:
+            logger.info("[TEARDOWN] Waiting for source pods before topology restore...")
+            kubectl_helper.wait_for_pods_ready(source_cluster_id, timeout=300)
+            self.cleanup_downstream_only_collection(c_after)
+            logger.info("[TEARDOWN] Restoring A→B topology...")
+            try:
+                switchover_helper(source_cluster_id, target_cluster_id)
+            except Exception as e:
+                logger.warning(f"switchover restore failed: {e}")
+            self.cleanup_resources()
+            self.log_test_end("test_force_promote_basic", True, time.time() - start_time)
+
+    def test_force_promote_during_target_restart(
+        self,
+        upstream_client,
+        downstream_client,
+        sync_timeout,
+        switchover_helper,
+        kubectl_helper,
+        source_cluster_id,
+        target_cluster_id,
+        upstream_uri,
+        upstream_token,
+        downstream_uri,
+        downstream_token,
+        pchannel_num,
+    ):
+        """Promote downstream while downstream pods are bouncing.
+
+        Mirrors snippets test_restart_b_during_force_promote.py: promote runs
+        async, restart happens in main thread mid-promote.
+        """
+        start_time = time.time()
+        c_name = self.gen_unique_name("test_fp_target_restart", max_length=50)
+        c_after = f"{c_name}_after"
+
+        self.log_test_start(
+            "test_force_promote_during_target_restart",
+            "FORCE_PROMOTE_TARGET_RESTART",
+            c_name,
+        )
+        self._upstream_client = upstream_client
+        self._downstream_client = downstream_client
+        self.resources_to_cleanup.append(("collection", c_name))
+        self.resources_to_cleanup.append(("collection", c_after))
+
+        try:
+            self.cleanup_collection(upstream_client, c_name)
+            schema = self.create_manual_id_schema(upstream_client)
+            upstream_client.create_collection(collection_name=c_name, schema=schema)
+            index_params = upstream_client.prepare_index_params()
+            index_params.add_index(
+                field_name="vector",
+                index_type="HNSW",
+                metric_type="L2",
+                params={"M": 16, "efConstruction": 200},
+            )
+            upstream_client.create_index(c_name, index_params)
+            upstream_client.load_collection(c_name)
+            assert self.wait_for_sync(
+                lambda: downstream_client.has_collection(c_name),
+                sync_timeout,
+                f"create collection {c_name}",
+            )
+
+            data = self.generate_test_data_with_id(DEFAULT_INSERT_COUNT, start_id=0)
+            upstream_client.insert(c_name, data)
+            upstream_client.flush(c_name)
+
+            def check_synced():
+                try:
+                    res = downstream_client.query(collection_name=c_name, filter="", output_fields=["count(*)"])
+                    return (res[0]["count(*)"] if res else 0) >= DEFAULT_INSERT_COUNT
+                except Exception:
+                    return False
+
+            assert self.wait_for_sync(check_synced, sync_timeout, f"sync {c_name}")
+
+            # Promote async — first call almost certainly fails because target dies
+            promote_err = []
+
+            def promote_thread():
+                try:
+                    self.do_force_promote(
+                        downstream_client,
+                        target_cluster_id,
+                        downstream_uri,
+                        downstream_token,
+                        pchannel_num,
+                    )
+                except Exception as e:
+                    promote_err.append(e)
+
+            th = threading.Thread(target=promote_thread, daemon=True)
+            th.start()
+
+            # Mid-promote: kill target pods, wait for them to come back
+            time.sleep(2)
+            logger.info("[FAILOVER] Killing target pods mid-promote...")
+            kubectl_helper.delete_pods(target_cluster_id)
+            time.sleep(2)
+            kubectl_helper.wait_for_pods_ready(target_cluster_id, timeout=300)
+
+            th.join(timeout=FAILOVER_TIMEOUT)
+            assert not th.is_alive(), "Background force_promote thread did not finish in time"
+            if promote_err:
+                logger.info(f"[EXPECTED] async promote raised: {promote_err[0]}")
+
+            # Retry until writable
+            self.promote_call_until_writable(
+                downstream_client,
+                target_cluster_id,
+                downstream_uri,
+                downstream_token,
+                pchannel_num,
+            )
+
+            schema_after = self.create_manual_id_schema(downstream_client)
+            downstream_client.create_collection(collection_name=c_after, schema=schema_after)
+            idx_after = downstream_client.prepare_index_params()
+            idx_after.add_index(
+                field_name="vector",
+                index_type="HNSW",
+                metric_type="L2",
+                params={"M": 16, "efConstruction": 200},
+            )
+            downstream_client.create_index(c_after, idx_after)
+            downstream_client.load_collection(c_after)
+            data_after = self.generate_test_data_with_id(100, start_id=0)
+            downstream_client.insert(c_after, data_after)
+            downstream_client.flush(c_after)
+            res = downstream_client.query(collection_name=c_after, filter="", output_fields=["count(*)"])
+            assert res and res[0]["count(*)"] >= 100, f"downstream not writable after target restart: {res}"
+        finally:
+            logger.info("[TEARDOWN] Waiting for target pods before topology restore...")
+            kubectl_helper.wait_for_pods_ready(target_cluster_id, timeout=300)
+            self.cleanup_downstream_only_collection(c_after)
+            logger.info("[TEARDOWN] Restoring A→B topology...")
+            try:
+                switchover_helper(source_cluster_id, target_cluster_id)
+            except Exception as e:
+                logger.warning(f"switchover restore failed: {e}")
+            self.cleanup_resources()
+            self.log_test_end(
+                "test_force_promote_during_target_restart",
+                True,
+                time.time() - start_time,
+            )
+
+    def test_force_promote_during_source_restart(
+        self,
+        upstream_client,
+        downstream_client,
+        sync_timeout,
+        switchover_helper,
+        kubectl_helper,
+        source_cluster_id,
+        target_cluster_id,
+        upstream_uri,
+        upstream_token,
+        downstream_uri,
+        downstream_token,
+        pchannel_num,
+    ):
+        """Promote downstream while upstream pods are bouncing.
+
+        Mirrors snippets test_restart_a_during_force_promote.py.
+        """
+        start_time = time.time()
+        c_name = self.gen_unique_name("test_fp_source_restart", max_length=50)
+        c_after = f"{c_name}_after"
+
+        self.log_test_start(
+            "test_force_promote_during_source_restart",
+            "FORCE_PROMOTE_SOURCE_RESTART",
+            c_name,
+        )
+        self._upstream_client = upstream_client
+        self._downstream_client = downstream_client
+        self.resources_to_cleanup.append(("collection", c_name))
+        self.resources_to_cleanup.append(("collection", c_after))
+
+        try:
+            self.cleanup_collection(upstream_client, c_name)
+            schema = self.create_manual_id_schema(upstream_client)
+            upstream_client.create_collection(collection_name=c_name, schema=schema)
+            index_params = upstream_client.prepare_index_params()
+            index_params.add_index(
+                field_name="vector",
+                index_type="HNSW",
+                metric_type="L2",
+                params={"M": 16, "efConstruction": 200},
+            )
+            upstream_client.create_index(c_name, index_params)
+            upstream_client.load_collection(c_name)
+            assert self.wait_for_sync(
+                lambda: downstream_client.has_collection(c_name),
+                sync_timeout,
+                f"create collection {c_name}",
+            )
+
+            data = self.generate_test_data_with_id(DEFAULT_INSERT_COUNT, start_id=0)
+            upstream_client.insert(c_name, data)
+            upstream_client.flush(c_name)
+
+            def check_synced():
+                try:
+                    res = downstream_client.query(collection_name=c_name, filter="", output_fields=["count(*)"])
+                    return (res[0]["count(*)"] if res else 0) >= DEFAULT_INSERT_COUNT
+                except Exception:
+                    return False
+
+            assert self.wait_for_sync(check_synced, sync_timeout, f"sync {c_name}")
+
+            promote_err = []
+
+            def promote_thread():
+                try:
+                    self.do_force_promote(
+                        downstream_client,
+                        target_cluster_id,
+                        downstream_uri,
+                        downstream_token,
+                        pchannel_num,
+                    )
+                except Exception as e:
+                    promote_err.append(e)
+
+            th = threading.Thread(target=promote_thread, daemon=True)
+            th.start()
+
+            time.sleep(2)
+            logger.info("[FAILOVER] Killing source pods mid-promote...")
+            kubectl_helper.delete_pods(source_cluster_id)
+            # Don't wait-ready here — promotion should succeed independently
+            # of whether the old primary is back.
+
+            th.join(timeout=FAILOVER_TIMEOUT)
+            assert not th.is_alive(), "Background force_promote thread did not finish in time"
+            if promote_err:
+                logger.info(f"[INFO] async promote raised (may be expected): {promote_err[0]}")
+
+            self.promote_call_until_writable(
+                downstream_client,
+                target_cluster_id,
+                downstream_uri,
+                downstream_token,
+                pchannel_num,
+            )
+
+            schema_after = self.create_manual_id_schema(downstream_client)
+            downstream_client.create_collection(collection_name=c_after, schema=schema_after)
+            idx_after = downstream_client.prepare_index_params()
+            idx_after.add_index(
+                field_name="vector",
+                index_type="HNSW",
+                metric_type="L2",
+                params={"M": 16, "efConstruction": 200},
+            )
+            downstream_client.create_index(c_after, idx_after)
+            downstream_client.load_collection(c_after)
+            data_after = self.generate_test_data_with_id(100, start_id=0)
+            downstream_client.insert(c_after, data_after)
+            downstream_client.flush(c_after)
+            res = downstream_client.query(collection_name=c_after, filter="", output_fields=["count(*)"])
+            assert res and res[0]["count(*)"] >= 100, f"downstream not writable after source restart: {res}"
+        finally:
+            logger.info("[TEARDOWN] Waiting for source pods before topology restore...")
+            kubectl_helper.wait_for_pods_ready(source_cluster_id, timeout=300)
+            self.cleanup_downstream_only_collection(c_after)
+            logger.info("[TEARDOWN] Restoring A→B topology...")
+            try:
+                switchover_helper(source_cluster_id, target_cluster_id)
+            except Exception as e:
+                logger.warning(f"switchover restore failed: {e}")
+            self.cleanup_resources()
+            self.log_test_end(
+                "test_force_promote_during_source_restart",
+                True,
+                time.time() - start_time,
+            )
+
+    def test_force_promote_with_incomplete_ddl(
+        self,
+        upstream_client,
+        downstream_client,
+        sync_timeout,
+        switchover_helper,
+        source_cluster_id,
+        target_cluster_id,
+        upstream_uri,
+        upstream_token,
+        downstream_uri,
+        downstream_token,
+        pchannel_num,
+    ):
+        """release_collection on upstream then immediately force_promote.
+
+        Exercises the Ignore=true path in
+        ackCallbackScheduler.fixIncompleteBroadcastsForForcePromote — DDL that
+        was mid-broadcast when the operator promoted gets marked Ignore=true
+        and resubmitted with replicate header cleared.
+
+        Mirrors snippets test_incomplete_broadcast_ddl.py.
+        """
+        start_time = time.time()
+        c_name = self.gen_unique_name("test_fp_inc_ddl", max_length=50)
+        c_after = f"{c_name}_after"
+
+        self.log_test_start(
+            "test_force_promote_with_incomplete_ddl",
+            "FORCE_PROMOTE_INCOMPLETE_DDL",
+            c_name,
+        )
+        self._upstream_client = upstream_client
+        self._downstream_client = downstream_client
+        self.resources_to_cleanup.append(("collection", c_name))
+        self.resources_to_cleanup.append(("collection", c_after))
+
+        try:
+            self.cleanup_collection(upstream_client, c_name)
+            schema = self.create_manual_id_schema(upstream_client)
+            upstream_client.create_collection(collection_name=c_name, schema=schema)
+            index_params = upstream_client.prepare_index_params()
+            index_params.add_index(
+                field_name="vector",
+                index_type="HNSW",
+                metric_type="L2",
+                params={"M": 16, "efConstruction": 200},
+            )
+            upstream_client.create_index(c_name, index_params)
+            upstream_client.load_collection(c_name)
+            assert self.wait_for_sync(
+                lambda: downstream_client.has_collection(c_name),
+                sync_timeout,
+                f"create collection {c_name}",
+            )
+
+            data = self.generate_test_data_with_id(DEFAULT_INSERT_COUNT, start_id=0)
+            upstream_client.insert(c_name, data)
+            upstream_client.flush(c_name)
+
+            def check_synced():
+                try:
+                    res = downstream_client.query(collection_name=c_name, filter="", output_fields=["count(*)"])
+                    return (res[0]["count(*)"] if res else 0) >= DEFAULT_INSERT_COUNT
+                except Exception:
+                    return False
+
+            assert self.wait_for_sync(check_synced, sync_timeout, f"sync {c_name}")
+
+            # In-flight DDL on upstream — may not finish replicating before promote
+            upstream_client.release_collection(c_name)
+            logger.info(f"[INCOMPLETE_DDL] release_collection sent on {c_name}")
+
+            # Immediately promote downstream (no sleep)
+            self.promote_call_until_writable(
+                downstream_client,
+                target_cluster_id,
+                downstream_uri,
+                downstream_token,
+                pchannel_num,
+            )
+
+            # On downstream (now primary), create new collection + insert
+            schema_after = self.create_manual_id_schema(downstream_client)
+            downstream_client.create_collection(collection_name=c_after, schema=schema_after)
+            idx_after = downstream_client.prepare_index_params()
+            idx_after.add_index(
+                field_name="vector",
+                index_type="HNSW",
+                metric_type="L2",
+                params={"M": 16, "efConstruction": 200},
+            )
+            downstream_client.create_index(c_after, idx_after)
+            downstream_client.load_collection(c_after)
+            data_after = self.generate_test_data_with_id(100, start_id=0)
+            downstream_client.insert(c_after, data_after)
+            downstream_client.flush(c_after)
+            res = downstream_client.query(collection_name=c_after, filter="", output_fields=["count(*)"])
+            assert res and res[0]["count(*)"] >= 100, f"downstream not writable after incomplete DDL: {res}"
+        finally:
+            self.cleanup_downstream_only_collection(c_after)
+            logger.info("[TEARDOWN] Restoring A→B topology...")
+            try:
+                switchover_helper(source_cluster_id, target_cluster_id)
+            except Exception as e:
+                logger.warning(f"switchover restore failed: {e}")
+            self.cleanup_resources()
+            self.log_test_end(
+                "test_force_promote_with_incomplete_ddl",
+                True,
+                time.time() - start_time,
+            )
+
+    def _build_partition_manifest(self, source_cluster_id, target_cluster_id, milvus_ns):
+        """Return a NetworkChaos dict that bidirectionally partitions source ↔ target."""
+        return {
+            "apiVersion": "chaos-mesh.org/v1alpha1",
+            "kind": "NetworkChaos",
+            "metadata": {
+                "name": f"{source_cluster_id}-failover-partition",
+                "namespace": milvus_ns,
+            },
+            "spec": {
+                "action": "partition",
+                "mode": "all",
+                "selector": {
+                    "namespaces": [milvus_ns],
+                    "labelSelectors": {"app.kubernetes.io/instance": source_cluster_id},
+                },
+                "direction": "both",
+                "target": {
+                    "mode": "all",
+                    "selector": {
+                        "namespaces": [milvus_ns],
+                        "labelSelectors": {"app.kubernetes.io/instance": target_cluster_id},
+                    },
+                },
+            },
+        }
+
+    def test_force_promote_with_network_partition(
+        self,
+        upstream_client,
+        downstream_client,
+        sync_timeout,
+        switchover_helper,
+        milvus_ns,
+        source_cluster_id,
+        target_cluster_id,
+        upstream_uri,
+        upstream_token,
+        downstream_uri,
+        downstream_token,
+        pchannel_num,
+    ):
+        """Apply NetworkChaos partition between source/target, then force_promote.
+
+        K8s-native scenario that the snippets tests can't reproduce. Simulates
+        the realistic emergency that motivates force_promote: secondary cannot
+        reach primary, operator force-promotes secondary.
+        """
+        start_time = time.time()
+        c_name = self.gen_unique_name("test_fp_partition", max_length=50)
+        c_after = f"{c_name}_after"
+        chaos_name = f"{source_cluster_id}-failover-partition"
+
+        self.log_test_start(
+            "test_force_promote_with_network_partition",
+            "FORCE_PROMOTE_PARTITION",
+            c_name,
+        )
+        self._upstream_client = upstream_client
+        self._downstream_client = downstream_client
+        self.resources_to_cleanup.append(("collection", c_name))
+        self.resources_to_cleanup.append(("collection", c_after))
+
+        chaos_path = None
+        try:
+            self.cleanup_collection(upstream_client, c_name)
+            schema = self.create_manual_id_schema(upstream_client)
+            upstream_client.create_collection(collection_name=c_name, schema=schema)
+            index_params = upstream_client.prepare_index_params()
+            index_params.add_index(
+                field_name="vector",
+                index_type="HNSW",
+                metric_type="L2",
+                params={"M": 16, "efConstruction": 200},
+            )
+            upstream_client.create_index(c_name, index_params)
+            upstream_client.load_collection(c_name)
+            assert self.wait_for_sync(
+                lambda: downstream_client.has_collection(c_name),
+                sync_timeout,
+                f"create collection {c_name}",
+            )
+
+            data = self.generate_test_data_with_id(DEFAULT_INSERT_COUNT, start_id=0)
+            upstream_client.insert(c_name, data)
+            upstream_client.flush(c_name)
+
+            def check_synced():
+                try:
+                    res = downstream_client.query(collection_name=c_name, filter="", output_fields=["count(*)"])
+                    return (res[0]["count(*)"] if res else 0) >= DEFAULT_INSERT_COUNT
+                except Exception:
+                    return False
+
+            assert self.wait_for_sync(check_synced, sync_timeout, f"sync {c_name}")
+
+            # Apply NetworkChaos
+            manifest = self._build_partition_manifest(source_cluster_id, target_cluster_id, milvus_ns)
+            with tempfile.NamedTemporaryFile(mode="w", suffix=".yaml", delete=False) as f:
+                yaml.safe_dump(manifest, f)
+                chaos_path = f.name
+            logger.info(f"[CHAOS] applying NetworkChaos partition: {chaos_path}")
+            apply_result = subprocess.run(
+                ["kubectl", "apply", "-f", chaos_path],
+                capture_output=True,
+                text=True,
+                check=False,
+            )
+            assert apply_result.returncode == 0, f"kubectl apply failed: {apply_result.stderr}"
+
+            # Let the partition propagate
+            time.sleep(10)
+
+            # Force promote downstream — it can't reach upstream
+            self.promote_call_until_writable(
+                downstream_client,
+                target_cluster_id,
+                downstream_uri,
+                downstream_token,
+                pchannel_num,
+            )
+
+            # Verify writable
+            schema_after = self.create_manual_id_schema(downstream_client)
+            downstream_client.create_collection(collection_name=c_after, schema=schema_after)
+            idx_after = downstream_client.prepare_index_params()
+            idx_after.add_index(
+                field_name="vector",
+                index_type="HNSW",
+                metric_type="L2",
+                params={"M": 16, "efConstruction": 200},
+            )
+            downstream_client.create_index(c_after, idx_after)
+            downstream_client.load_collection(c_after)
+            data_after = self.generate_test_data_with_id(100, start_id=0)
+            downstream_client.insert(c_after, data_after)
+            downstream_client.flush(c_after)
+            res = downstream_client.query(collection_name=c_after, filter="", output_fields=["count(*)"])
+            assert res and res[0]["count(*)"] >= 100, f"downstream not writable after partition+promote: {res}"
+        finally:
+            # Remove the partition first so switchover can fan-out across clusters
+            logger.info(f"[CHAOS] cleaning up NetworkChaos {chaos_name}")
+            subprocess.run(
+                ["kubectl", "delete", "networkchaos", chaos_name, "-n", milvus_ns],
+                capture_output=True,
+                text=True,
+                check=False,
+            )
+            if chaos_path and os.path.exists(chaos_path):
+                os.unlink(chaos_path)
+            # Allow partition to clear before switchover
+            time.sleep(10)
+            self.cleanup_downstream_only_collection(c_after)
+            logger.info("[TEARDOWN] Restoring A→B topology...")
+            try:
+                switchover_helper(source_cluster_id, target_cluster_id)
+            except Exception as e:
+                logger.warning(f"switchover restore failed: {e}")
+            self.cleanup_resources()
+            self.log_test_end(
+                "test_force_promote_with_network_partition",
+                True,
+                time.time() - start_time,
+            )
+
+    def _do_one_endurance_iteration(
+        self,
+        iteration,
+        upstream_client,
+        downstream_client,
+        sync_timeout,
+        switchover_helper,
+        source_cluster_id,
+        target_cluster_id,
+        upstream_uri,
+        upstream_token,
+        downstream_uri,
+        downstream_token,
+        pchannel_num,
+    ):
+        """One endurance iteration: DDL + insert + force_promote + DDL on new primary."""
+        c_before = f"endurance_before_{iteration}"
+        c_after = f"endurance_after_{iteration}"
+
+        # Reset to A→B topology (in case prior iteration left state behind)
+        try:
+            switchover_helper(source_cluster_id, target_cluster_id)
+        except Exception as e:
+            raise RuntimeError(f"[iter {iteration}] pre-iter switchover failed") from e
+
+        # DDL before failover on upstream
+        self.cleanup_collection(upstream_client, c_before)
+        schema = self.create_manual_id_schema(upstream_client)
+        upstream_client.create_collection(collection_name=c_before, schema=schema)
+        idx = upstream_client.prepare_index_params()
+        idx.add_index(
+            field_name="vector",
+            index_type="HNSW",
+            metric_type="L2",
+            params={"M": 16, "efConstruction": 200},
+        )
+        upstream_client.create_index(c_before, idx)
+        upstream_client.load_collection(c_before)
+        assert self.wait_for_sync(
+            lambda: downstream_client.has_collection(c_before),
+            sync_timeout,
+            f"[iter {iteration}] create {c_before}",
+        )
+        data = self.generate_test_data_with_id(100, start_id=0)
+        upstream_client.insert(c_before, data)
+        upstream_client.flush(c_before)
+
+        def check_synced():
+            try:
+                res = downstream_client.query(collection_name=c_before, filter="", output_fields=["count(*)"])
+                return (res[0]["count(*)"] if res else 0) >= 100
+            except Exception:
+                return False
+
+        assert self.wait_for_sync(check_synced, sync_timeout, f"[iter {iteration}] sync {c_before}")
+
+        # In-flight DDL
+        upstream_client.release_collection(c_before)
+
+        # Force promote
+        self.promote_call_until_writable(
+            downstream_client,
+            target_cluster_id,
+            downstream_uri,
+            downstream_token,
+            pchannel_num,
+        )
+
+        # DDL after failover on downstream
+        schema_after = self.create_manual_id_schema(downstream_client)
+        downstream_client.create_collection(collection_name=c_after, schema=schema_after)
+        idx_after = downstream_client.prepare_index_params()
+        idx_after.add_index(
+            field_name="vector",
+            index_type="HNSW",
+            metric_type="L2",
+            params={"M": 16, "efConstruction": 200},
+        )
+        downstream_client.create_index(c_after, idx_after)
+        downstream_client.load_collection(c_after)
+        data_after = self.generate_test_data_with_id(100, start_id=0)
+        downstream_client.insert(c_after, data_after)
+        downstream_client.flush(c_after)
+        res = downstream_client.query(collection_name=c_after, filter="", output_fields=["count(*)"])
+        assert res and res[0]["count(*)"] >= 100, f"[iter {iteration}] downstream count check failed: {res}"
+
+        # Cleanup both
+        try:
+            downstream_client.drop_collection(c_after)
+        except Exception as e:
+            logger.warning(f"[iter {iteration}] drop {c_after} on downstream: {e}")
+        try:
+            downstream_client.drop_collection(c_before)
+        except Exception as e:
+            logger.warning(f"[iter {iteration}] drop {c_before} on downstream: {e}")
+
+    @pytest.mark.skipif(
+        not os.getenv("RUN_ENDURANCE"),
+        reason="Endurance test only runs when RUN_ENDURANCE is set",
+    )
+    def test_endurance_force_promote(
+        self,
+        upstream_client,
+        downstream_client,
+        sync_timeout,
+        switchover_helper,
+        source_cluster_id,
+        target_cluster_id,
+        upstream_uri,
+        upstream_token,
+        downstream_uri,
+        downstream_token,
+        pchannel_num,
+    ):
+        """Endurance loop: repeated force_promote + DDL for ENDURANCE_DURATION_MINUTES."""
+        duration_minutes = int(os.getenv("ENDURANCE_DURATION_MINUTES", "30"))
+        deadline = time.time() + duration_minutes * 60
+        iteration = 0
+        start_time = time.time()
+        self.log_test_start("test_endurance_force_promote", "ENDURANCE_FORCE_PROMOTE", "")
+        self._upstream_client = upstream_client
+        self._downstream_client = downstream_client
+
+        try:
+            while time.time() < deadline:
+                iteration += 1
+                logger.info(f"=== Endurance iteration {iteration} ===")
+                self._do_one_endurance_iteration(
+                    iteration,
+                    upstream_client,
+                    downstream_client,
+                    sync_timeout,
+                    switchover_helper,
+                    source_cluster_id,
+                    target_cluster_id,
+                    upstream_uri,
+                    upstream_token,
+                    downstream_uri,
+                    downstream_token,
+                    pchannel_num,
+                )
+            logger.info(f"PASSED endurance: {iteration} iterations in {duration_minutes}m")
+        finally:
+            logger.info("[TEARDOWN] Restoring A→B topology after endurance...")
+            try:
+                switchover_helper(source_cluster_id, target_cluster_id)
+            except Exception as e:
+                logger.warning(f"switchover restore failed: {e}")
+            self.log_test_end("test_endurance_force_promote", True, time.time() - start_time)

tests/python_client/cdc/testcases/test_force_promote_cleanup.py

@@ -0,0 +1,39 @@
+import importlib
+import importlib.util
+import sys
+import types
+
+if importlib.util.find_spec("yaml") is None:
+    yaml = types.ModuleType("yaml")
+    yaml.safe_dump = lambda *args, **kwargs: None
+    sys.modules.setdefault("yaml", yaml)
+
+_TestCDCForcePromote = importlib.import_module("cdc.testcases.test_force_promote").TestCDCForcePromote
+
+
+class FakeClient:
+    def __init__(self):
+        self.collections = {"downstream_only"}
+        self.dropped = []
+
+    def has_collection(self, collection_name):
+        return collection_name in self.collections
+
+    def drop_collection(self, collection_name):
+        self.dropped.append(collection_name)
+        self.collections.discard(collection_name)
+
+
+def test_cleanup_downstream_only_collection_before_topology_restore():
+    test = _TestCDCForcePromote()
+    test.resources_to_cleanup = [
+        ("collection", "normal_collection"),
+        ("collection", "downstream_only"),
+    ]
+    test._downstream_client = FakeClient()
+
+    test.cleanup_downstream_only_collection("downstream_only")
+
+    assert test._downstream_client.dropped == ["downstream_only"]
+    assert ("collection", "downstream_only") not in test.resources_to_cleanup
+    assert ("collection", "normal_collection") in test.resources_to_cleanup

tests/python_client/cdc/testcases/test_fts_and_text.py

@@ -0,0 +1,715 @@
+"""
+CDC sync tests for full-text search (BM25), text match, and phrase match operations.
+"""
+
+import random
+import time
+
+import pytest
+from pymilvus import AnnSearchRequest, DataType, RRFRanker
+
+from .base import TestCDCSyncBase, logger
+
+
+class TestCDCSyncFTSAndText(TestCDCSyncBase):
+    """Test CDC sync for full-text search, text match, and phrase match operations."""
+
+    def setup_method(self):
+        """Setup for each test method."""
+        self.resources_to_cleanup = []
+
+    def teardown_method(self):
+        """Cleanup after each test method - only cleanup upstream, downstream will sync."""
+        upstream_client = getattr(self, "_upstream_client", None)
+
+        if upstream_client:
+            for resource_type, resource_name in self.resources_to_cleanup:
+                if resource_type == "collection":
+                    self.cleanup_collection(upstream_client, resource_name)
+
+            time.sleep(1)  # Allow cleanup to sync to downstream
+
+    # -------------------------------------------------------------------------
+    # Test 1: FTS insert and search replication
+    # -------------------------------------------------------------------------
+
+    @pytest.mark.parametrize("analyzer_type", ["standard", "english"])
+    def test_fts_insert_and_search(
+        self,
+        upstream_client,
+        downstream_client,
+        sync_timeout,
+        analyzer_type,
+    ):
+        """Test FTS (BM25) insert and search replication.
+
+        Creates an FTS collection with BM25 function, inserts 200 docs, creates
+        SPARSE_INVERTED_INDEX (BM25) and HNSW indexes, then verifies that FTS search
+        results replicate to downstream with sufficient overlap.
+        """
+        start_time = time.time()
+        collection_name = self.gen_unique_name("fts_search", max_length=50)
+
+        self.log_test_start(
+            "test_fts_insert_and_search",
+            f"FTS_INSERT_SEARCH(analyzer={analyzer_type})",
+            collection_name,
+        )
+
+        self._upstream_client = upstream_client
+        self.resources_to_cleanup.append(("collection", collection_name))
+
+        try:
+            # Initial cleanup
+            self.cleanup_collection(upstream_client, collection_name)
+
+            # Create FTS schema and collection
+            logger.info(f"[CREATE] Creating FTS collection '{collection_name}' with analyzer_type='{analyzer_type}'")
+            schema = self.create_fts_schema(upstream_client, analyzer_type)
+            upstream_client.create_collection(collection_name, schema=schema)
+
+            # Insert 200 documents
+            fts_data = self.generate_fts_data(200)
+            logger.info(f"[INSERT] Inserting {len(fts_data)} FTS documents upstream")
+            result = upstream_client.insert(collection_name, fts_data)
+            inserted_count = result.get("insert_count", len(fts_data))
+            logger.info(f"[INSERT] Inserted {inserted_count} documents")
+
+            upstream_client.flush(collection_name)
+
+            # Create SPARSE_INVERTED_INDEX on sparse_output (BM25)
+            index_params = upstream_client.prepare_index_params()
+            index_params.add_index(
+                field_name="sparse_output",
+                index_type="SPARSE_INVERTED_INDEX",
+                metric_type="BM25",
+                params={"bm25_k1": 1.5, "bm25_b": 0.75},
+            )
+            # Create HNSW on dense_vector
+            index_params.add_index(
+                field_name="dense_vector",
+                index_type="HNSW",
+                metric_type="L2",
+                params={"M": 8, "efConstruction": 64},
+            )
+            upstream_client.create_index(collection_name, index_params)
+            upstream_client.load_collection(collection_name)
+
+            # Run FTS search on upstream
+            query_text = "vector database similarity search"
+            logger.info(f"[SEARCH] Running FTS search upstream with query: '{query_text}'")
+            upstream_results = upstream_client.search(
+                collection_name,
+                data=[query_text],
+                anns_field="sparse_output",
+                limit=10,
+                search_params={"metric_type": "BM25"},
+                output_fields=["text_field", "category"],
+            )
+            upstream_ids = set()
+            if upstream_results and len(upstream_results) > 0:
+                for hit in upstream_results[0]:
+                    upstream_ids.add(hit.get("id") or hit.id)
+
+            logger.info(f"[SEARCH] Upstream FTS returned {len(upstream_ids)} results")
+
+            # Wait for collection to appear downstream
+            def check_collection_exists():
+                return downstream_client.has_collection(collection_name)
+
+            assert self.wait_for_sync(
+                check_collection_exists,
+                sync_timeout,
+                f"collection '{collection_name}' creation sync",
+            ), f"Collection '{collection_name}' did not sync to downstream"
+
+            # Wait for data + index to sync and downstream search results overlap
+            def check_fts_overlap():
+                try:
+                    ds_results = downstream_client.search(
+                        collection_name,
+                        data=[query_text],
+                        anns_field="sparse_output",
+                        limit=10,
+                        search_params={"metric_type": "BM25"},
+                        output_fields=["text_field", "category"],
+                    )
+                    if not ds_results or len(ds_results) == 0:
+                        return False
+                    downstream_ids = set()
+                    for hit in ds_results[0]:
+                        downstream_ids.add(hit.get("id") or hit.id)
+                    if not downstream_ids:
+                        return False
+                    if len(upstream_ids) == 0:
+                        return len(downstream_ids) > 0
+                    overlap = len(upstream_ids & downstream_ids) / max(len(upstream_ids), 1)
+                    logger.info(
+                        f"[OVERLAP] FTS search overlap: {overlap:.2f} "
+                        f"(upstream={len(upstream_ids)}, downstream={len(downstream_ids)})"
+                    )
+                    return overlap >= self.SEARCH_OVERLAP_THRESHOLD
+                except Exception as e:
+                    logger.warning(f"FTS overlap check failed: {e}")
+                    return False
+
+            assert self.wait_for_sync(
+                check_fts_overlap,
+                sync_timeout,
+                f"FTS search overlap sync (analyzer={analyzer_type})",
+            ), f"FTS search results did not reach overlap threshold {self.SEARCH_OVERLAP_THRESHOLD} on downstream"
+
+            duration = time.time() - start_time
+            self.log_test_end("test_fts_insert_and_search", True, duration)
+
+        except Exception as exc:
+            duration = time.time() - start_time
+            self.log_test_end("test_fts_insert_and_search", False, duration)
+            raise exc
+
+    # -------------------------------------------------------------------------
+    # Test 2: TEXT_MATCH sync
+    # -------------------------------------------------------------------------
+
+    @pytest.mark.parametrize("analyzer_type", ["standard", "english"])
+    def test_text_match_sync(
+        self,
+        upstream_client,
+        downstream_client,
+        sync_timeout,
+        analyzer_type,
+    ):
+        """Test TEXT_MATCH query replication.
+
+        Creates a collection with a VARCHAR field that has analyzer + match enabled,
+        inserts 100 rows, and verifies that TEXT_MATCH queries return the same count
+        on both upstream and downstream.
+        """
+        start_time = time.time()
+        collection_name = self.gen_unique_name("text_match", max_length=50)
+
+        self.log_test_start(
+            "test_text_match_sync",
+            f"TEXT_MATCH(analyzer={analyzer_type})",
+            collection_name,
+        )
+
+        self._upstream_client = upstream_client
+        self.resources_to_cleanup.append(("collection", collection_name))
+
+        try:
+            self.cleanup_collection(upstream_client, collection_name)
+
+            # Build schema with analyzer-enabled VARCHAR
+            schema = upstream_client.create_schema(enable_dynamic_field=False)
+            schema.add_field("id", DataType.INT64, is_primary=True, auto_id=True)
+            schema.add_field("dense_vector", DataType.FLOAT_VECTOR, dim=128)
+            schema.add_field(
+                "text_field",
+                DataType.VARCHAR,
+                max_length=2048,
+                enable_analyzer=True,
+                enable_match=True,
+                analyzer_params={"type": analyzer_type},
+            )
+
+            logger.info(f"[CREATE] Creating text-match collection '{collection_name}'")
+            upstream_client.create_collection(collection_name, schema=schema)
+
+            # Insert 100 rows from FTS_SENTENCES
+            data = []
+            for i in range(100):
+                data.append(
+                    {
+                        "dense_vector": [random.random() for _ in range(128)],
+                        "text_field": self.FTS_SENTENCES[i % len(self.FTS_SENTENCES)],
+                    }
+                )
+
+            logger.info(f"[INSERT] Inserting {len(data)} rows upstream")
+            upstream_client.insert(collection_name, data)
+            upstream_client.flush(collection_name)
+
+            # Create HNSW index and load
+            index_params = upstream_client.prepare_index_params()
+            index_params.add_index(
+                field_name="dense_vector",
+                index_type="HNSW",
+                metric_type="L2",
+                params={"M": 8, "efConstruction": 64},
+            )
+            upstream_client.create_index(collection_name, index_params)
+            upstream_client.load_collection(collection_name)
+
+            # Run TEXT_MATCH query on upstream
+            filter_expr = "TEXT_MATCH(text_field, 'vector database')"
+            logger.info(f"[QUERY] Upstream TEXT_MATCH query: {filter_expr}")
+            upstream_results = upstream_client.query(
+                collection_name,
+                filter=filter_expr,
+                output_fields=["id", "text_field"],
+                limit=100,
+            )
+            upstream_count = len(upstream_results)
+            logger.info(f"[QUERY] Upstream TEXT_MATCH returned {upstream_count} rows")
+
+            # Wait for collection to appear downstream
+            def check_collection_exists():
+                return downstream_client.has_collection(collection_name)
+
+            assert self.wait_for_sync(
+                check_collection_exists,
+                sync_timeout,
+                f"collection '{collection_name}' creation sync",
+            )
+
+            # Wait for downstream query count to match
+            def check_count_match():
+                try:
+                    ds_results = downstream_client.query(
+                        collection_name,
+                        filter=filter_expr,
+                        output_fields=["id", "text_field"],
+                        limit=100,
+                    )
+                    ds_count = len(ds_results)
+                    logger.info(f"[VERIFY] TEXT_MATCH downstream count={ds_count}, upstream count={upstream_count}")
+                    return ds_count == upstream_count
+                except Exception as e:
+                    logger.warning(f"TEXT_MATCH count check failed: {e}")
+                    return False
+
+            assert self.wait_for_sync(
+                check_count_match,
+                sync_timeout,
+                f"TEXT_MATCH query count sync (analyzer={analyzer_type})",
+            ), f"TEXT_MATCH query count mismatch between upstream ({upstream_count}) and downstream after timeout"
+
+            duration = time.time() - start_time
+            self.log_test_end("test_text_match_sync", True, duration)
+
+        except Exception as exc:
+            duration = time.time() - start_time
+            self.log_test_end("test_text_match_sync", False, duration)
+            raise exc
+
+    # -------------------------------------------------------------------------
+    # Test 3: PHRASE_MATCH sync
+    # -------------------------------------------------------------------------
+
+    @pytest.mark.parametrize("analyzer_type", ["standard", "english"])
+    def test_phrase_match_sync(
+        self,
+        upstream_client,
+        downstream_client,
+        sync_timeout,
+        analyzer_type,
+    ):
+        """Test PHRASE_MATCH query replication.
+
+        Same collection setup as text_match.  Queries PHRASE_MATCH with slop=1 and
+        verifies that upstream and downstream return the same count.
+        """
+        start_time = time.time()
+        collection_name = self.gen_unique_name("phrase_match", max_length=50)
+
+        self.log_test_start(
+            "test_phrase_match_sync",
+            f"PHRASE_MATCH(analyzer={analyzer_type})",
+            collection_name,
+        )
+
+        self._upstream_client = upstream_client
+        self.resources_to_cleanup.append(("collection", collection_name))
+
+        try:
+            self.cleanup_collection(upstream_client, collection_name)
+
+            # Build schema with analyzer-enabled VARCHAR
+            schema = upstream_client.create_schema(enable_dynamic_field=False)
+            schema.add_field("id", DataType.INT64, is_primary=True, auto_id=True)
+            schema.add_field("dense_vector", DataType.FLOAT_VECTOR, dim=128)
+            schema.add_field(
+                "text_field",
+                DataType.VARCHAR,
+                max_length=2048,
+                enable_analyzer=True,
+                enable_match=True,
+                analyzer_params={"type": analyzer_type},
+            )
+
+            logger.info(f"[CREATE] Creating phrase-match collection '{collection_name}'")
+            upstream_client.create_collection(collection_name, schema=schema)
+
+            # Insert 100 rows from FTS_SENTENCES
+            data = []
+            for i in range(100):
+                data.append(
+                    {
+                        "dense_vector": [random.random() for _ in range(128)],
+                        "text_field": self.FTS_SENTENCES[i % len(self.FTS_SENTENCES)],
+                    }
+                )
+
+            logger.info(f"[INSERT] Inserting {len(data)} rows upstream")
+            upstream_client.insert(collection_name, data)
+            upstream_client.flush(collection_name)
+
+            # Create HNSW index and load
+            index_params = upstream_client.prepare_index_params()
+            index_params.add_index(
+                field_name="dense_vector",
+                index_type="HNSW",
+                metric_type="L2",
+                params={"M": 8, "efConstruction": 64},
+            )
+            upstream_client.create_index(collection_name, index_params)
+            upstream_client.load_collection(collection_name)
+
+            # Run PHRASE_MATCH query on upstream (slop=1)
+            filter_expr = "PHRASE_MATCH(text_field, 'brown fox', 1)"
+            logger.info(f"[QUERY] Upstream PHRASE_MATCH query: {filter_expr}")
+            upstream_results = upstream_client.query(
+                collection_name,
+                filter=filter_expr,
+                output_fields=["id", "text_field"],
+                limit=100,
+            )
+            upstream_count = len(upstream_results)
+            logger.info(f"[QUERY] Upstream PHRASE_MATCH returned {upstream_count} rows")
+
+            # Wait for collection to appear downstream
+            def check_collection_exists():
+                return downstream_client.has_collection(collection_name)
+
+            assert self.wait_for_sync(
+                check_collection_exists,
+                sync_timeout,
+                f"collection '{collection_name}' creation sync",
+            )
+
+            # Wait for downstream query count to match
+            def check_count_match():
+                try:
+                    ds_results = downstream_client.query(
+                        collection_name,
+                        filter=filter_expr,
+                        output_fields=["id", "text_field"],
+                        limit=100,
+                    )
+                    ds_count = len(ds_results)
+                    logger.info(f"[VERIFY] PHRASE_MATCH downstream count={ds_count}, upstream count={upstream_count}")
+                    return ds_count == upstream_count
+                except Exception as e:
+                    logger.warning(f"PHRASE_MATCH count check failed: {e}")
+                    return False
+
+            assert self.wait_for_sync(
+                check_count_match,
+                sync_timeout,
+                f"PHRASE_MATCH query count sync (analyzer={analyzer_type})",
+            ), f"PHRASE_MATCH query count mismatch between upstream ({upstream_count}) and downstream after timeout"
+
+            duration = time.time() - start_time
+            self.log_test_end("test_phrase_match_sync", True, duration)
+
+        except Exception as exc:
+            duration = time.time() - start_time
+            self.log_test_end("test_phrase_match_sync", False, duration)
+            raise exc
+
+    # -------------------------------------------------------------------------
+    # Test 4: Hybrid search (FTS + dense) replication
+    # -------------------------------------------------------------------------
+
+    def test_hybrid_search_fts_dense(
+        self,
+        upstream_client,
+        downstream_client,
+        sync_timeout,
+    ):
+        """Test hybrid search (BM25 sparse + HNSW dense) replication.
+
+        Creates an FTS collection, inserts 300 docs, builds both sparse (BM25) and
+        dense (HNSW) indexes, and verifies that hybrid search results on downstream
+        have sufficient overlap with upstream results.
+        """
+        start_time = time.time()
+        collection_name = self.gen_unique_name("hybrid_fts", max_length=50)
+
+        self.log_test_start(
+            "test_hybrid_search_fts_dense",
+            "HYBRID_SEARCH_FTS_DENSE",
+            collection_name,
+        )
+
+        self._upstream_client = upstream_client
+        self.resources_to_cleanup.append(("collection", collection_name))
+
+        try:
+            self.cleanup_collection(upstream_client, collection_name)
+
+            # Create FTS schema (standard analyzer)
+            schema = self.create_fts_schema(upstream_client, "standard")
+            logger.info(f"[CREATE] Creating hybrid-search collection '{collection_name}'")
+            upstream_client.create_collection(collection_name, schema=schema)
+
+            # Insert 300 documents
+            fts_data = self.generate_fts_data(300)
+            logger.info(f"[INSERT] Inserting {len(fts_data)} documents upstream")
+            upstream_client.insert(collection_name, fts_data)
+            upstream_client.flush(collection_name)
+
+            # Create both indexes
+            index_params = upstream_client.prepare_index_params()
+            index_params.add_index(
+                field_name="sparse_output",
+                index_type="SPARSE_INVERTED_INDEX",
+                metric_type="BM25",
+                params={"bm25_k1": 1.5, "bm25_b": 0.75},
+            )
+            index_params.add_index(
+                field_name="dense_vector",
+                index_type="HNSW",
+                metric_type="L2",
+                params={"M": 8, "efConstruction": 64},
+            )
+            upstream_client.create_index(collection_name, index_params)
+            upstream_client.load_collection(collection_name)
+
+            # Build hybrid search requests
+            sparse_req = AnnSearchRequest(
+                data=["vector database similarity search"],
+                anns_field="sparse_output",
+                param={"metric_type": "BM25"},
+                limit=10,
+            )
+            dense_query_vec = [random.random() for _ in range(128)]
+            dense_req = AnnSearchRequest(
+                data=[dense_query_vec],
+                anns_field="dense_vector",
+                param={"metric_type": "L2", "params": {"ef": 64}},
+                limit=10,
+            )
+
+            logger.info("[SEARCH] Running hybrid search on upstream")
+            upstream_results = upstream_client.hybrid_search(
+                collection_name,
+                reqs=[sparse_req, dense_req],
+                ranker=RRFRanker(),
+                limit=10,
+                output_fields=["text_field", "category"],
+            )
+
+            upstream_ids = set()
+            if upstream_results and len(upstream_results) > 0:
+                for hit in upstream_results[0]:
+                    upstream_ids.add(hit.get("id") or hit.id)
+
+            logger.info(f"[SEARCH] Upstream hybrid search returned {len(upstream_ids)} results")
+
+            # Wait for collection to appear downstream
+            def check_collection_exists():
+                return downstream_client.has_collection(collection_name)
+
+            assert self.wait_for_sync(
+                check_collection_exists,
+                sync_timeout,
+                f"collection '{collection_name}' creation sync",
+            )
+
+            # Wait for downstream hybrid search overlap
+            def check_hybrid_overlap():
+                try:
+                    ds_sparse_req = AnnSearchRequest(
+                        data=["vector database similarity search"],
+                        anns_field="sparse_output",
+                        param={"metric_type": "BM25"},
+                        limit=10,
+                    )
+                    ds_dense_req = AnnSearchRequest(
+                        data=[dense_query_vec],
+                        anns_field="dense_vector",
+                        param={"metric_type": "L2", "params": {"ef": 64}},
+                        limit=10,
+                    )
+                    ds_results = downstream_client.hybrid_search(
+                        collection_name,
+                        reqs=[ds_sparse_req, ds_dense_req],
+                        ranker=RRFRanker(),
+                        limit=10,
+                        output_fields=["text_field", "category"],
+                    )
+                    if not ds_results or len(ds_results) == 0:
+                        return False
+                    downstream_ids = set()
+                    for hit in ds_results[0]:
+                        downstream_ids.add(hit.get("id") or hit.id)
+                    if not downstream_ids:
+                        return False
+                    if len(upstream_ids) == 0:
+                        return len(downstream_ids) > 0
+                    overlap = len(upstream_ids & downstream_ids) / max(len(upstream_ids), 1)
+                    logger.info(
+                        f"[OVERLAP] Hybrid search overlap: {overlap:.2f} "
+                        f"(upstream={len(upstream_ids)}, downstream={len(downstream_ids)})"
+                    )
+                    return overlap >= self.SEARCH_OVERLAP_THRESHOLD
+                except Exception as e:
+                    logger.warning(f"Hybrid overlap check failed: {e}")
+                    return False
+
+            assert self.wait_for_sync(
+                check_hybrid_overlap,
+                sync_timeout,
+                "hybrid search (FTS + dense) overlap sync",
+            ), f"Hybrid search results did not reach overlap threshold {self.SEARCH_OVERLAP_THRESHOLD} on downstream"
+
+            duration = time.time() - start_time
+            self.log_test_end("test_hybrid_search_fts_dense", True, duration)
+
+        except Exception as exc:
+            duration = time.time() - start_time
+            self.log_test_end("test_hybrid_search_fts_dense", False, duration)
+            raise exc
+
+    # -------------------------------------------------------------------------
+    # Test 5: FTS after switchover
+    # -------------------------------------------------------------------------
+
+    def test_fts_after_switchover(
+        self,
+        upstream_client,
+        downstream_client,
+        sync_timeout,
+        switchover_helper,
+        source_cluster_id,
+        target_cluster_id,
+    ):
+        """Test FTS replication continues correctly after CDC topology switchover.
+
+        1. Create FTS collection, insert 100 docs, build index, verify sync.
+        2. Perform switchover so downstream becomes the new source.
+        3. Insert 50 more docs into the new source (original downstream).
+        4. Verify FTS search works on the new downstream (original upstream).
+        5. Switch back to original topology.
+        """
+        start_time = time.time()
+        collection_name = self.gen_unique_name("fts_switchover", max_length=50)
+
+        self.log_test_start(
+            "test_fts_after_switchover",
+            "FTS_AFTER_SWITCHOVER",
+            collection_name,
+        )
+
+        self._upstream_client = upstream_client
+        self.resources_to_cleanup.append(("collection", collection_name))
+
+        try:
+            self.cleanup_collection(upstream_client, collection_name)
+
+            # Phase 1: Create FTS collection and index on original upstream
+            logger.info(f"[PHASE1] Creating FTS collection '{collection_name}' on upstream")
+            schema = self.create_fts_schema(upstream_client, "standard")
+            upstream_client.create_collection(collection_name, schema=schema)
+
+            fts_data = self.generate_fts_data(100)
+            logger.info(f"[PHASE1] Inserting {len(fts_data)} documents upstream")
+            upstream_client.insert(collection_name, fts_data)
+            upstream_client.flush(collection_name)
+
+            index_params = upstream_client.prepare_index_params()
+            index_params.add_index(
+                field_name="sparse_output",
+                index_type="SPARSE_INVERTED_INDEX",
+                metric_type="BM25",
+                params={"bm25_k1": 1.5, "bm25_b": 0.75},
+            )
+            index_params.add_index(
+                field_name="dense_vector",
+                index_type="HNSW",
+                metric_type="L2",
+                params={"M": 8, "efConstruction": 64},
+            )
+            upstream_client.create_index(collection_name, index_params)
+            upstream_client.load_collection(collection_name)
+
+            # Verify initial sync to downstream
+            def check_initial_sync():
+                if not downstream_client.has_collection(collection_name):
+                    return False
+                try:
+                    ds_results = downstream_client.search(
+                        collection_name,
+                        data=["vector database"],
+                        anns_field="sparse_output",
+                        limit=5,
+                        search_params={"metric_type": "BM25"},
+                        output_fields=["text_field"],
+                    )
+                    return ds_results is not None and len(ds_results) > 0
+                except Exception:
+                    return False
+
+            assert self.wait_for_sync(
+                check_initial_sync,
+                sync_timeout,
+                f"initial FTS sync for '{collection_name}'",
+            ), f"Initial FTS sync failed for collection '{collection_name}'"
+
+            logger.info("[PHASE1] Initial FTS sync verified")
+
+            # Phase 2: Switchover — downstream becomes new source
+            logger.info(f"[PHASE2] Switching CDC direction: {target_cluster_id} -> {source_cluster_id}")
+            switchover_helper(target_cluster_id, source_cluster_id)
+
+            # Insert 50 more docs to the new source (original downstream)
+            extra_data = self.generate_fts_data(50)
+            logger.info(f"[PHASE2] Inserting {len(extra_data)} additional docs to new source (downstream_client)")
+            downstream_client.insert(collection_name, extra_data)
+            downstream_client.flush(collection_name)
+
+            # Phase 3: Verify FTS search works on new downstream (original upstream)
+            query_text = "distributed database replication"
+
+            def check_fts_on_new_downstream():
+                try:
+                    results = upstream_client.search(
+                        collection_name,
+                        data=[query_text],
+                        anns_field="sparse_output",
+                        limit=5,
+                        search_params={"metric_type": "BM25"},
+                        output_fields=["text_field"],
+                    )
+                    return results is not None and len(results) > 0
+                except Exception as e:
+                    logger.warning(f"FTS on new downstream check failed: {e}")
+                    return False
+
+            assert self.wait_for_sync(
+                check_fts_on_new_downstream,
+                sync_timeout,
+                "FTS search on new downstream after switchover",
+            ), "FTS search on new downstream (original upstream) failed after switchover"
+
+            logger.info("[PHASE3] FTS search verified on new downstream after switchover")
+
+            # Phase 4: Switch back to original topology
+            logger.info(f"[PHASE4] Switching back to original topology: {source_cluster_id} -> {target_cluster_id}")
+            switchover_helper(source_cluster_id, target_cluster_id)
+
+            duration = time.time() - start_time
+            self.log_test_end("test_fts_after_switchover", True, duration)
+
+        except Exception as exc:
+            # Best-effort restore original topology on failure
+            try:
+                logger.warning("[RECOVER] Attempting to restore original CDC topology after failure")
+                switchover_helper(source_cluster_id, target_cluster_id)
+            except Exception as restore_exc:
+                logger.error(f"[RECOVER] Failed to restore topology: {restore_exc}")
+            duration = time.time() - start_time
+            self.log_test_end("test_fts_after_switchover", False, duration)
+            raise exc

tests/python_client/cdc/testcases/test_import_2pc.py

@@ -0,0 +1,52 @@
+import pytest
+from chaos.checker import Import2PCChecker
+from common import common_func as cf
+from common.common_type import CaseLabel
+from pymilvus import connections
+
+
+class TestCDCImport2PC:
+    @pytest.mark.tags(CaseLabel.CDC)
+    def test_import_2pc_manual_commit_replicates_to_downstream(
+        self,
+        upstream_client,
+        upstream_uri,
+        upstream_token,
+        downstream_uri,
+        downstream_token,
+        import_2pc_minio_endpoint,
+        import_2pc_minio_bucket,
+        import_2pc_downstream_minio_endpoint,
+        import_2pc_downstream_minio_bucket,
+        import_2pc_rows,
+        sync_timeout,
+    ):
+        """
+        target: verify Import 2PC manual commit is a valid CDC workload
+        method: create an upstream manual import job with auto_commit=false, stage the same parquet files in both
+                object stores, wait for primary and secondary Uncommitted, then CommitImport on upstream
+        expected: imported PKs stay invisible on both clusters before commit and become visible on both clusters
+                  after the replicated CommitImport is consumed
+        """
+        collection_name = cf.gen_unique_str("CDCImport2PC_")
+        connections.connect("default", uri=upstream_uri, token=upstream_token)
+        checker = Import2PCChecker(
+            collection_name=collection_name,
+            rows_per_import=import_2pc_rows,
+            minio_endpoint=import_2pc_minio_endpoint,
+            bucket_name=import_2pc_minio_bucket,
+            uri=upstream_uri,
+            token=upstream_token,
+            downstream_uri=downstream_uri,
+            downstream_token=downstream_token,
+            downstream_minio_endpoint=import_2pc_downstream_minio_endpoint,
+            downstream_bucket_name=import_2pc_downstream_minio_bucket,
+            visibility_timeout=max(sync_timeout, 180),
+        )
+        try:
+            res, ok = checker.run_task()
+            assert ok, res
+        finally:
+            checker.terminate()
+            if upstream_client.has_collection(collection_name):
+                upstream_client.drop_collection(collection_name)

tests/python_client/cdc/testcases/test_index.py

@@ -0,0 +1,731 @@
+"""
+CDC sync tests for index operations.
+"""
+
+import time
+import random
+import pytest
+from common.common_type import CaseLabel
+from .base import TestCDCSyncBase, logger
+
+
+@pytest.mark.tags(CaseLabel.CDC)
+class TestCDCSyncIndex(TestCDCSyncBase):
+    """Test CDC sync for index operations."""
+
+    def setup_method(self):
+        """Setup for each test method."""
+        self.resources_to_cleanup = []
+
+    def teardown_method(self):
+        """Cleanup after each test method - only cleanup upstream, downstream will sync."""
+        upstream_client = getattr(self, "_upstream_client", None)
+
+        if upstream_client:
+            for resource_type, resource_name in self.resources_to_cleanup:
+                if resource_type == "collection":
+                    self.cleanup_collection(upstream_client, resource_name)
+
+            time.sleep(1)  # Allow cleanup to sync to downstream
+
+    def test_create_index(self, upstream_client, downstream_client, sync_timeout):
+        """Test CREATE_INDEX operation sync."""
+        # Store upstream client for teardown
+        self._upstream_client = upstream_client
+
+        collection_name = self.gen_unique_name("test_col_create_idx")
+        self.resources_to_cleanup.append(("collection", collection_name))
+
+        # Initial cleanup
+        self.cleanup_collection(upstream_client, collection_name)
+
+        # Create collection
+        upstream_client.create_collection(
+            collection_name=collection_name,
+            schema=self.create_default_schema(upstream_client),
+        )
+
+        # Wait for creation to sync
+        def check_create():
+            return downstream_client.has_collection(collection_name)
+
+        assert self.wait_for_sync(
+            check_create, sync_timeout, f"create collection {collection_name}"
+        )
+
+        # Create index
+        index_params = upstream_client.prepare_index_params()
+        index_params.add_index(
+            field_name="vector",
+            index_type="IVF_FLAT",
+            metric_type="L2",
+            params={"nlist": 128},
+        )
+        upstream_client.create_index(collection_name, index_params)
+
+        # Wait for index creation to sync
+        def check_index():
+            try:
+                downstream_indexes = downstream_client.list_indexes(collection_name)
+                return len(downstream_indexes) > 0
+            except:
+                return False
+
+        assert self.wait_for_sync(
+            check_index, sync_timeout, f"create index on {collection_name}"
+        )
+
+    def test_drop_index(self, upstream_client, downstream_client, sync_timeout):
+        """Test DROP_INDEX operation sync."""
+        # Store upstream client for teardown
+        self._upstream_client = upstream_client
+
+        collection_name = self.gen_unique_name("test_col_drop_idx")
+        self.resources_to_cleanup.append(("collection", collection_name))
+
+        # Initial cleanup
+        self.cleanup_collection(upstream_client, collection_name)
+
+        # Create collection and index
+        upstream_client.create_collection(
+            collection_name=collection_name,
+            schema=self.create_default_schema(upstream_client),
+        )
+
+        index_params = upstream_client.prepare_index_params()
+        index_params.add_index(
+            field_name="vector",
+            index_type="IVF_FLAT",
+            metric_type="L2",
+            params={"nlist": 128},
+        )
+        upstream_client.create_index(collection_name, index_params)
+
+        # Wait for setup to sync
+        def check_setup():
+            try:
+                return (
+                    downstream_client.has_collection(collection_name)
+                    and len(downstream_client.list_indexes(collection_name)) > 0
+                )
+            except:
+                return False
+
+        assert self.wait_for_sync(
+            check_setup, sync_timeout, f"setup collection and index {collection_name}"
+        )
+
+        # Drop index
+        upstream_client.drop_index(collection_name, "vector")
+
+        # Wait for index drop to sync
+        def check_drop():
+            try:
+                downstream_indexes = downstream_client.list_indexes(collection_name)
+                return len(downstream_indexes) == 0
+            except:
+                return True  # If error, assume index is dropped
+
+        assert self.wait_for_sync(
+            check_drop, sync_timeout, f"drop index on {collection_name}"
+        )
+
+    def test_create_vector_indexes_comprehensive(
+        self, upstream_client, downstream_client, sync_timeout
+    ):
+        """Test CREATE_INDEX operation sync for all vector index types."""
+        # Store upstream client for teardown
+        self._upstream_client = upstream_client
+
+        # Test cases for different vector types and their applicable indexes
+        test_cases = [
+            # FLOAT_VECTOR indexes
+            {
+                "field_name": "float_vector",
+                "field_type": "FLOAT_VECTOR",
+                "index_tests": [
+                    {"index_type": "FLAT", "metric_type": "L2", "params": {}},
+                    {
+                        "index_type": "IVF_FLAT",
+                        "metric_type": "L2",
+                        "params": {"nlist": 128},
+                    },
+                    {
+                        "index_type": "IVF_SQ8",
+                        "metric_type": "L2",
+                        "params": {"nlist": 128},
+                    },
+                    {
+                        "index_type": "IVF_PQ",
+                        "metric_type": "L2",
+                        "params": {"nlist": 128, "m": 16, "nbits": 8},
+                    },
+                    {
+                        "index_type": "HNSW",
+                        "metric_type": "L2",
+                        "params": {"M": 16, "efConstruction": 200},
+                    },
+                ],
+            },
+            # FLOAT16_VECTOR indexes
+            {
+                "field_name": "float16_vector",
+                "field_type": "FLOAT16_VECTOR",
+                "index_tests": [
+                    {"index_type": "FLAT", "metric_type": "L2", "params": {}},
+                    {
+                        "index_type": "IVF_FLAT",
+                        "metric_type": "L2",
+                        "params": {"nlist": 128},
+                    },
+                    {
+                        "index_type": "HNSW",
+                        "metric_type": "L2",
+                        "params": {"M": 16, "efConstruction": 200},
+                    },
+                ],
+            },
+            # BINARY_VECTOR indexes
+            {
+                "field_name": "binary_vector",
+                "field_type": "BINARY_VECTOR",
+                "index_tests": [
+                    {"index_type": "BIN_FLAT", "metric_type": "HAMMING", "params": {}},
+                    {
+                        "index_type": "BIN_IVF_FLAT",
+                        "metric_type": "HAMMING",
+                        "params": {"nlist": 128},
+                    },
+                ],
+            },
+            # SPARSE_FLOAT_VECTOR indexes
+            {
+                "field_name": "sparse_vector",
+                "field_type": "SPARSE_FLOAT_VECTOR",
+                "index_tests": [
+                    {
+                        "index_type": "SPARSE_INVERTED_INDEX",
+                        "metric_type": "IP",
+                        "params": {},
+                    },
+                ],
+            },
+        ]
+
+        for test_case in test_cases:
+            for index_test in test_case["index_tests"]:
+                collection_name = self.gen_unique_name(
+                    f"test_idx_{test_case['field_type'].lower()}_{index_test['index_type'].lower()}"
+                )
+                self.resources_to_cleanup.append(("collection", collection_name))
+
+                try:
+                    logger.info(
+                        f"[INDEX_TEST] Testing {test_case['field_type']} with {index_test['index_type']} index"
+                    )
+
+                    # Initial cleanup
+                    self.cleanup_collection(upstream_client, collection_name)
+
+                    # Create collection with specific vector field
+                    schema = self._create_vector_schema(
+                        upstream_client,
+                        test_case["field_name"],
+                        test_case["field_type"],
+                    )
+                    upstream_client.create_collection(
+                        collection_name=collection_name, schema=schema
+                    )
+
+                    # Wait for creation to sync
+                    def check_create():
+                        return downstream_client.has_collection(collection_name)
+
+                    assert self.wait_for_sync(
+                        check_create,
+                        sync_timeout,
+                        f"create collection {collection_name}",
+                    )
+
+                    # Insert test data before creating index (better practice)
+                    test_data = self._generate_test_data_for_vector_field(
+                        test_case["field_name"], test_case["field_type"], 100
+                    )
+                    upstream_client.insert(collection_name, test_data)
+                    upstream_client.flush(collection_name)
+                    logger.info(
+                        f"[DATA_INSERTED] Inserted 100 records before creating {test_case['field_type']} index"
+                    )
+
+                    # Create specific index
+                    index_params = upstream_client.prepare_index_params()
+                    index_params.add_index(
+                        field_name=test_case["field_name"],
+                        index_type=index_test["index_type"],
+                        metric_type=index_test["metric_type"],
+                        params=index_test["params"],
+                    )
+                    upstream_client.create_index(collection_name, index_params)
+
+                    # Wait for index creation to sync
+                    def check_index():
+                        try:
+                            downstream_indexes = downstream_client.list_indexes(
+                                collection_name
+                            )
+                            return len(downstream_indexes) > 0
+                        except:
+                            return False
+
+                    assert self.wait_for_sync(
+                        check_index,
+                        sync_timeout,
+                        f"create {index_test['index_type']} index on {collection_name}",
+                    )
+
+                    # Verify index details
+                    try:
+                        index_info = downstream_client.describe_index(
+                            collection_name, test_case["field_name"]
+                        )
+                        logger.info(
+                            f"[INDEX_VERIFICATION] {index_test['index_type']} index created successfully: {index_info}"
+                        )
+                    except Exception as e:
+                        logger.warning(
+                            f"Failed to verify {index_test['index_type']} index details: {e}"
+                        )
+
+                except Exception as e:
+                    logger.error(
+                        f"[INDEX_ERROR] Failed to test {test_case['field_type']} with {index_test['index_type']}: {e}"
+                    )
+                    raise
+
+    def test_create_scalar_indexes_comprehensive(
+        self, upstream_client, downstream_client, sync_timeout
+    ):
+        """Test CREATE_INDEX operation sync for all scalar field index types."""
+        # Store upstream client for teardown
+        self._upstream_client = upstream_client
+
+        # Test cases for different scalar types and their applicable indexes
+        test_cases = [
+            # VARCHAR indexes
+            {
+                "field_name": "varchar_field",
+                "field_type": "VARCHAR",
+                "index_tests": [
+                    {"index_type": "INVERTED", "params": {}},
+                    {"index_type": "Trie", "params": {}},
+                ],
+            },
+            # BOOL indexes
+            {
+                "field_name": "bool_field",
+                "field_type": "BOOL",
+                "index_tests": [
+                    {"index_type": "INVERTED", "params": {}},
+                ],
+            },
+            # INT32 indexes
+            {
+                "field_name": "int32_field",
+                "field_type": "INT32",
+                "index_tests": [
+                    {"index_type": "INVERTED", "params": {}},
+                    {"index_type": "STL_SORT", "params": {}},
+                ],
+            },
+            # INT64 indexes
+            {
+                "field_name": "int64_field",
+                "field_type": "INT64",
+                "index_tests": [
+                    {"index_type": "INVERTED", "params": {}},
+                    {"index_type": "STL_SORT", "params": {}},
+                ],
+            },
+            # FLOAT indexes
+            {
+                "field_name": "float_field",
+                "field_type": "FLOAT",
+                "index_tests": [
+                    {"index_type": "INVERTED", "params": {}},
+                ],
+            },
+            # DOUBLE indexes
+            {
+                "field_name": "double_field",
+                "field_type": "DOUBLE",
+                "index_tests": [
+                    {"index_type": "INVERTED", "params": {}},
+                ],
+            },
+            # ARRAY indexes
+            {
+                "field_name": "int32_array",
+                "field_type": "ARRAY",
+                "element_type": "INT32",
+                "index_tests": [
+                    {"index_type": "INVERTED", "params": {}},
+                ],
+            },
+            # JSON indexes - use AUTOINDEX (recommended) with proper JSON path syntax
+            {
+                "field_name": "json_field",
+                "field_type": "JSON",
+                "index_tests": [
+                    {
+                        "index_type": "AUTOINDEX",
+                        "params": {
+                            "json_path": 'json_field["name"]',
+                            "json_cast_type": "VARCHAR",
+                        },
+                    },
+                ],
+            },
+        ]
+
+        for test_case in test_cases:
+            for index_test in test_case["index_tests"]:
+                collection_name = self.gen_unique_name(
+                    f"test_idx_{test_case['field_type'].lower()}_{index_test['index_type'].lower()}"
+                )
+                self.resources_to_cleanup.append(("collection", collection_name))
+
+                try:
+                    logger.info(
+                        f"[SCALAR_INDEX_TEST] Testing {test_case['field_type']} with {index_test['index_type']} index"
+                    )
+
+                    # Initial cleanup
+                    self.cleanup_collection(upstream_client, collection_name)
+
+                    # Create collection with specific scalar field
+                    schema = self._create_scalar_schema(upstream_client, test_case)
+                    upstream_client.create_collection(
+                        collection_name=collection_name, schema=schema
+                    )
+
+                    # Wait for creation to sync
+                    def check_create():
+                        return downstream_client.has_collection(collection_name)
+
+                    assert self.wait_for_sync(
+                        check_create,
+                        sync_timeout,
+                        f"create collection {collection_name}",
+                    )
+
+                    # Insert test data before creating index (better practice)
+                    test_data = self._generate_test_data_for_scalar_field(
+                        test_case["field_name"], test_case["field_type"], 100
+                    )
+                    upstream_client.insert(collection_name, test_data)
+                    upstream_client.flush(collection_name)
+                    logger.info(
+                        f"[DATA_INSERTED] Inserted 100 records before creating {test_case['field_type']} index"
+                    )
+
+                    # Create specific index
+                    index_params = upstream_client.prepare_index_params()
+                    if test_case["field_type"] == "JSON":
+                        # JSON fields need special handling with index_name
+                        index_params.add_index(
+                            field_name=test_case["field_name"],
+                            index_type=index_test["index_type"],
+                            index_name=f"{test_case['field_name']}_name_index",
+                            params=index_test["params"],
+                        )
+                    else:
+                        index_params.add_index(
+                            field_name=test_case["field_name"],
+                            index_type=index_test["index_type"],
+                            params=index_test["params"],
+                        )
+                    upstream_client.create_index(collection_name, index_params)
+
+                    # Wait for index creation to sync
+                    def check_index():
+                        try:
+                            downstream_indexes = downstream_client.list_indexes(
+                                collection_name
+                            )
+                            return len(downstream_indexes) > 0
+                        except:
+                            return False
+
+                    assert self.wait_for_sync(
+                        check_index,
+                        sync_timeout,
+                        f"create {index_test['index_type']} index on {collection_name}",
+                    )
+
+                    # Verify index details
+                    try:
+                        index_info = downstream_client.describe_index(
+                            collection_name, test_case["field_name"]
+                        )
+                        logger.info(
+                            f"[SCALAR_INDEX_VERIFICATION] {index_test['index_type']} index created successfully: {index_info}"
+                        )
+                    except Exception as e:
+                        logger.warning(
+                            f"Failed to verify {index_test['index_type']} scalar index details: {e}"
+                        )
+
+                except Exception as e:
+                    logger.error(
+                        f"[SCALAR_INDEX_ERROR] Failed to test {test_case['field_type']} with {index_test['index_type']}: {e}"
+                    )
+                    raise
+
+    def _create_vector_schema(self, client, field_name, field_type):
+        """Create schema for vector index testing."""
+        from pymilvus import DataType
+
+        schema = client.create_schema(enable_dynamic_field=True)
+        schema.add_field("id", DataType.INT64, is_primary=True, auto_id=True)
+
+        if field_type == "FLOAT_VECTOR":
+            schema.add_field(field_name, DataType.FLOAT_VECTOR, dim=128)
+        elif field_type == "FLOAT16_VECTOR":
+            schema.add_field(field_name, DataType.FLOAT16_VECTOR, dim=64)
+        elif field_type == "BFLOAT16_VECTOR":
+            schema.add_field(field_name, DataType.BFLOAT16_VECTOR, dim=64)
+        elif field_type == "BINARY_VECTOR":
+            schema.add_field(field_name, DataType.BINARY_VECTOR, dim=128)
+        elif field_type == "SPARSE_FLOAT_VECTOR":
+            schema.add_field(field_name, DataType.SPARSE_FLOAT_VECTOR)
+        elif field_type == "INT8_VECTOR":
+            schema.add_field(field_name, DataType.INT8_VECTOR, dim=128)
+
+        return schema
+
+    def _create_scalar_schema(self, client, test_case):
+        """Create schema for scalar index testing."""
+        from pymilvus import DataType
+
+        schema = client.create_schema(enable_dynamic_field=True)
+        schema.add_field("id", DataType.INT64, is_primary=True, auto_id=True)
+        schema.add_field(
+            "vector", DataType.FLOAT_VECTOR, dim=128
+        )  # Required for collection
+
+        field_name = test_case["field_name"]
+        field_type = test_case["field_type"]
+
+        if field_type == "VARCHAR":
+            schema.add_field(field_name, DataType.VARCHAR, max_length=1000)
+        elif field_type == "BOOL":
+            schema.add_field(field_name, DataType.BOOL)
+        elif field_type == "INT32":
+            schema.add_field(field_name, DataType.INT32)
+        elif field_type == "INT64":
+            schema.add_field(field_name, DataType.INT64)
+        elif field_type == "FLOAT":
+            schema.add_field(field_name, DataType.FLOAT)
+        elif field_type == "DOUBLE":
+            schema.add_field(field_name, DataType.DOUBLE)
+        elif field_type == "ARRAY" and test_case.get("element_type") == "INT32":
+            schema.add_field(
+                field_name,
+                DataType.ARRAY,
+                element_type=DataType.INT32,
+                max_capacity=100,
+            )
+        elif field_type == "JSON":
+            schema.add_field(field_name, DataType.JSON)
+
+        return schema
+
+    def _generate_test_data_for_vector_field(self, field_name, field_type, count=100):
+        """Generate test data for specific vector field type."""
+        from pymilvus import DataType
+
+        data = []
+        for _ in range(count):
+            record = {}
+
+            if field_type == "FLOAT_VECTOR":
+                vectors = self._gen_vectors(1, 128, DataType.FLOAT_VECTOR)
+                record[field_name] = vectors[0]
+            elif field_type == "FLOAT16_VECTOR":
+                vectors = self._gen_vectors(1, 64, DataType.FLOAT16_VECTOR)
+                record[field_name] = vectors[0]
+            elif field_type == "BFLOAT16_VECTOR":
+                vectors = self._gen_vectors(1, 64, DataType.BFLOAT16_VECTOR)
+                record[field_name] = vectors[0]
+            elif field_type == "BINARY_VECTOR":
+                vectors = self._gen_vectors(1, 128, DataType.BINARY_VECTOR)
+                record[field_name] = vectors[0]
+            elif field_type == "SPARSE_FLOAT_VECTOR":
+                vectors = self._gen_vectors(1, 1000, DataType.SPARSE_FLOAT_VECTOR)
+                record[field_name] = vectors[0]
+            elif field_type == "INT8_VECTOR":
+                vectors = self._gen_vectors(1, 128, DataType.INT8_VECTOR)
+                record[field_name] = vectors[0]
+
+            data.append(record)
+
+        return data
+
+    def _generate_test_data_for_scalar_field(self, field_name, field_type, count=100):
+        """Generate test data for specific scalar field type."""
+        data = []
+        for i in range(count):
+            record = {
+                "vector": [
+                    random.random() for _ in range(128)
+                ],  # Required base vector field
+            }
+
+            if field_type == "VARCHAR":
+                record[field_name] = f"test_string_{i}_{random.randint(1000, 9999)}"
+            elif field_type == "BOOL":
+                record[field_name] = random.choice([True, False])
+            elif field_type == "INT32":
+                record[field_name] = random.randint(-1000000, 1000000)
+            elif field_type == "INT64":
+                record[field_name] = random.randint(-1000000000, 1000000000)
+            elif field_type == "FLOAT":
+                record[field_name] = random.uniform(-1000.0, 1000.0)
+            elif field_type == "DOUBLE":
+                record[field_name] = random.uniform(-1000.0, 1000.0)
+            elif field_type == "ARRAY":
+                record[field_name] = [
+                    random.randint(-100, 100) for _ in range(random.randint(1, 10))
+                ]
+            elif field_type == "JSON":
+                record[field_name] = {
+                    "name": f"test_item_{i}",
+                    "value": random.randint(1, 1000),
+                    "category": random.choice(["A", "B", "C"]),
+                    "metadata": {
+                        "score": random.uniform(0.0, 100.0),
+                        "created": f"2024-01-{random.randint(1, 28):02d}",
+                    },
+                }
+
+            data.append(record)
+
+        return data
+
+    def test_create_bfloat16_int8_vector_indexes(
+        self, upstream_client, downstream_client, sync_timeout
+    ):
+        """Test CREATE_INDEX operation sync for BFLOAT16_VECTOR and INT8_VECTOR (combined test due to 4-vector limit)."""
+        # Store upstream client for teardown
+        self._upstream_client = upstream_client
+
+        # Test cases for BFLOAT16_VECTOR and INT8_VECTOR
+        test_cases = [
+            # BFLOAT16_VECTOR indexes - use AUTOINDEX for compatibility
+            {
+                "field_name": "bfloat16_vector",
+                "field_type": "BFLOAT16_VECTOR",
+                "index_tests": [
+                    {"index_type": "AUTOINDEX", "metric_type": "L2", "params": {}},
+                ],
+            },
+            # INT8_VECTOR indexes - use AUTOINDEX for compatibility
+            {
+                "field_name": "int8_vector",
+                "field_type": "INT8_VECTOR",
+                "index_tests": [
+                    {"index_type": "AUTOINDEX", "metric_type": "L2", "params": {}},
+                ],
+            },
+        ]
+
+        for test_case in test_cases:
+            for index_test in test_case["index_tests"]:
+                collection_name = self.gen_unique_name(
+                    f"test_idx_{test_case['field_type'].lower()}_{index_test['index_type'].lower()}"
+                )
+                self.resources_to_cleanup.append(("collection", collection_name))
+
+                try:
+                    logger.info(
+                        f"[{test_case['field_type']}_INDEX_TEST] Testing {test_case['field_type']} with {index_test['index_type']} index"
+                    )
+
+                    # Initial cleanup
+                    self.cleanup_collection(upstream_client, collection_name)
+
+                    # Create collection with specific vector field
+                    schema = self._create_vector_schema(
+                        upstream_client,
+                        test_case["field_name"],
+                        test_case["field_type"],
+                    )
+                    upstream_client.create_collection(
+                        collection_name=collection_name, schema=schema
+                    )
+
+                    # Wait for creation to sync
+                    def check_create():
+                        return downstream_client.has_collection(collection_name)
+
+                    assert self.wait_for_sync(
+                        check_create,
+                        sync_timeout,
+                        f"create collection {collection_name}",
+                    )
+
+                    # Insert test data before creating index (better practice)
+                    if test_case["field_type"] == "BFLOAT16_VECTOR":
+                        test_data = self.generate_bfloat16_test_data(100)
+                    else:  # INT8_VECTOR
+                        test_data = self.generate_int8_test_data(100)
+                    upstream_client.insert(collection_name, test_data)
+                    upstream_client.flush(collection_name)
+                    logger.info(
+                        f"[DATA_INSERTED] Inserted 100 records before creating {test_case['field_type']} index"
+                    )
+
+                    # Create specific index
+                    index_params = upstream_client.prepare_index_params()
+                    index_params.add_index(
+                        field_name=test_case["field_name"],
+                        index_type=index_test["index_type"],
+                        metric_type=index_test["metric_type"],
+                        params=index_test["params"],
+                    )
+                    upstream_client.create_index(collection_name, index_params)
+
+                    # Wait for index creation to sync
+                    def check_index():
+                        try:
+                            downstream_indexes = downstream_client.list_indexes(
+                                collection_name
+                            )
+                            return len(downstream_indexes) > 0
+                        except:
+                            return False
+
+                    assert self.wait_for_sync(
+                        check_index,
+                        sync_timeout,
+                        f"create {index_test['index_type']} index on {collection_name}",
+                    )
+
+                    # Verify index details
+                    try:
+                        index_info = downstream_client.describe_index(
+                            collection_name, test_case["field_name"]
+                        )
+                        logger.info(
+                            f"[{test_case['field_type']}_INDEX_VERIFICATION] {index_test['index_type']} index created successfully: {index_info}"
+                        )
+                    except Exception as e:
+                        logger.warning(
+                            f"Failed to verify {index_test['index_type']} {test_case['field_type']} index details: {e}"
+                        )
+
+                except Exception as e:
+                    logger.error(
+                        f"[{test_case['field_type']}_INDEX_ERROR] Failed to test {test_case['field_type']} with {index_test['index_type']}: {e}"
+                    )
+                    raise

tests/python_client/cdc/testcases/test_multi_database.py

@@ -0,0 +1,316 @@
+"""
+CDC sync tests for multi-database operations.
+"""
+
+import time
+
+from pymilvus import MilvusClient
+
+from .base import TestCDCSyncBase, logger
+
+
+class TestCDCSyncMultiDatabase(TestCDCSyncBase):
+    """Test CDC sync for operations across multiple databases."""
+
+    def setup_method(self):
+        """Setup for each test method."""
+        self.resources_to_cleanup = []
+
+    def teardown_method(self):
+        """Cleanup after each test method - only cleanup upstream, downstream will sync."""
+        upstream_uri = getattr(self, "_upstream_uri", None)
+        upstream_token = getattr(self, "_upstream_token", None)
+
+        if upstream_uri:
+            # Re-create a default-db client for cleanup
+            try:
+                client = MilvusClient(uri=upstream_uri, token=upstream_token)
+            except Exception as e:
+                logger.warning(f"[CLEANUP] Failed to create upstream client: {e}")
+                return
+
+            # Clean up collections in databases first, then databases
+            for resource_type, resource_data in self.resources_to_cleanup:
+                if resource_type == "collection_in_db":
+                    db_name, c_name = resource_data
+                    try:
+                        db_client = MilvusClient(
+                            uri=upstream_uri,
+                            token=upstream_token,
+                            db_name=db_name,
+                        )
+                        if db_client.has_collection(c_name):
+                            logger.info(f"[CLEANUP] Dropping collection {c_name} in db {db_name}")
+                            db_client.drop_collection(c_name)
+                        db_client.close()
+                    except Exception as e:
+                        logger.warning(f"[CLEANUP] Failed to drop collection {c_name} in db {db_name}: {e}")
+
+            for resource_type, resource_data in self.resources_to_cleanup:
+                if resource_type == "database":
+                    db_name = resource_data
+                    self.cleanup_database(client, db_name)
+
+            client.close()
+            time.sleep(1)  # Allow cleanup to sync to downstream
+
+    def test_create_collections_in_multiple_dbs(
+        self,
+        upstream_uri,
+        upstream_token,
+        downstream_uri,
+        downstream_token,
+        upstream_client,
+        downstream_client,
+        sync_timeout,
+    ):
+        """Test creating collections in multiple databases syncs to downstream."""
+        self._upstream_uri = upstream_uri
+        self._upstream_token = upstream_token
+
+        db_name_1 = self.gen_unique_name("test_mdb_db1")
+        db_name_2 = self.gen_unique_name("test_mdb_db2")
+        c_name_1 = self.gen_unique_name("test_mdb_col1")
+        c_name_2 = self.gen_unique_name("test_mdb_col2")
+
+        self.resources_to_cleanup.append(("collection_in_db", (db_name_1, c_name_1)))
+        self.resources_to_cleanup.append(("collection_in_db", (db_name_2, c_name_2)))
+        self.resources_to_cleanup.append(("database", db_name_1))
+        self.resources_to_cleanup.append(("database", db_name_2))
+
+        # Initial cleanup
+        self.cleanup_database(upstream_client, db_name_1)
+        self.cleanup_database(upstream_client, db_name_2)
+
+        # Create databases
+        upstream_client.create_database(db_name_1)
+        upstream_client.create_database(db_name_2)
+
+        # Create DB-scoped clients
+        up_db1_client = MilvusClient(uri=upstream_uri, token=upstream_token, db_name=db_name_1)
+        up_db2_client = MilvusClient(uri=upstream_uri, token=upstream_token, db_name=db_name_2)
+
+        # Create collection in db1 with 100 rows
+        schema1 = self.create_default_schema(up_db1_client)
+        up_db1_client.create_collection(collection_name=c_name_1, schema=schema1, consistency_level="Strong")
+        up_db1_client.insert(c_name_1, self.generate_test_data(100))
+        up_db1_client.flush(c_name_1)
+
+        # Create collection in db2 with 200 rows
+        schema2 = self.create_default_schema(up_db2_client)
+        up_db2_client.create_collection(collection_name=c_name_2, schema=schema2, consistency_level="Strong")
+        up_db2_client.insert(c_name_2, self.generate_test_data(200))
+        up_db2_client.flush(c_name_2)
+
+        up_db1_client.close()
+        up_db2_client.close()
+
+        # Wait for both databases and collections to appear on downstream
+        def check_db1_collection():
+            try:
+                if db_name_1 not in downstream_client.list_databases():
+                    return False
+                dn_db1 = MilvusClient(uri=downstream_uri, token=downstream_token, db_name=db_name_1)
+                exists = dn_db1.has_collection(c_name_1)
+                if exists:
+                    stats = dn_db1.get_collection_stats(c_name_1)
+                    logger.info(f"Downstream db1 collection stats: {stats}")
+                dn_db1.close()
+                return exists
+            except Exception as e:
+                logger.warning(f"Check db1 collection failed: {e}")
+                return False
+
+        assert self.wait_for_sync(check_db1_collection, sync_timeout, f"collection {c_name_1} in {db_name_1}")
+
+        def check_db2_collection():
+            try:
+                if db_name_2 not in downstream_client.list_databases():
+                    return False
+                dn_db2 = MilvusClient(uri=downstream_uri, token=downstream_token, db_name=db_name_2)
+                exists = dn_db2.has_collection(c_name_2)
+                if exists:
+                    stats = dn_db2.get_collection_stats(c_name_2)
+                    logger.info(f"Downstream db2 collection stats: {stats}")
+                dn_db2.close()
+                return exists
+            except Exception as e:
+                logger.warning(f"Check db2 collection failed: {e}")
+                return False
+
+        assert self.wait_for_sync(check_db2_collection, sync_timeout, f"collection {c_name_2} in {db_name_2}")
+
+        # Verify row counts
+        dn_db1 = MilvusClient(uri=downstream_uri, token=downstream_token, db_name=db_name_1)
+        dn_db2 = MilvusClient(uri=downstream_uri, token=downstream_token, db_name=db_name_2)
+        try:
+            stats1 = dn_db1.get_collection_stats(c_name_1)
+            stats2 = dn_db2.get_collection_stats(c_name_2)
+            logger.info(f"DB1 collection row count: {stats1.get('row_count')}")
+            logger.info(f"DB2 collection row count: {stats2.get('row_count')}")
+            assert stats1.get("row_count", 0) >= 100, (
+                f"Expected >= 100 rows in {c_name_1}, got {stats1.get('row_count')}"
+            )
+            assert stats2.get("row_count", 0) >= 200, (
+                f"Expected >= 200 rows in {c_name_2}, got {stats2.get('row_count')}"
+            )
+        finally:
+            dn_db1.close()
+            dn_db2.close()
+
+    def test_drop_db_with_collections(
+        self,
+        upstream_uri,
+        upstream_token,
+        downstream_uri,
+        downstream_token,
+        upstream_client,
+        downstream_client,
+        sync_timeout,
+    ):
+        """Test drop DB with collection syncs to downstream (DB gone from downstream)."""
+        self._upstream_uri = upstream_uri
+        self._upstream_token = upstream_token
+
+        db_name = self.gen_unique_name("test_mdb_drop_db")
+        c_name = self.gen_unique_name("test_mdb_drop_col")
+
+        self.resources_to_cleanup.append(("collection_in_db", (db_name, c_name)))
+        self.resources_to_cleanup.append(("database", db_name))
+
+        # Initial cleanup
+        self.cleanup_database(upstream_client, db_name)
+
+        # Create database and collection
+        upstream_client.create_database(db_name)
+        up_db_client = MilvusClient(uri=upstream_uri, token=upstream_token, db_name=db_name)
+        schema = self.create_default_schema(up_db_client)
+        up_db_client.create_collection(collection_name=c_name, schema=schema, consistency_level="Strong")
+        up_db_client.insert(c_name, self.generate_test_data(50))
+        up_db_client.flush(c_name)
+        up_db_client.close()
+
+        # Wait for DB + collection to sync to downstream
+        def check_created():
+            try:
+                if db_name not in downstream_client.list_databases():
+                    return False
+                dn = MilvusClient(uri=downstream_uri, token=downstream_token, db_name=db_name)
+                exists = dn.has_collection(c_name)
+                dn.close()
+                return exists
+            except Exception as e:
+                logger.warning(f"Check DB+collection created: {e}")
+                return False
+
+        assert self.wait_for_sync(check_created, sync_timeout, f"create db {db_name} with collection")
+
+        # Drop collection then DB in upstream
+        up_db_client2 = MilvusClient(uri=upstream_uri, token=upstream_token, db_name=db_name)
+        up_db_client2.drop_collection(c_name)
+        up_db_client2.close()
+        upstream_client.drop_database(db_name)
+
+        assert db_name not in upstream_client.list_databases(), (
+            f"Database {db_name} still exists in upstream after drop"
+        )
+
+        # Wait for DB to be gone on downstream
+        def check_db_dropped():
+            return db_name not in downstream_client.list_databases()
+
+        assert self.wait_for_sync(check_db_dropped, sync_timeout, f"drop database {db_name}")
+
+    def test_cross_db_operations(
+        self,
+        upstream_uri,
+        upstream_token,
+        downstream_uri,
+        downstream_token,
+        upstream_client,
+        downstream_client,
+        sync_timeout,
+    ):
+        """Test cross-DB operations: create DB, collection, insert, alter DB properties, create 2nd collection."""
+        self._upstream_uri = upstream_uri
+        self._upstream_token = upstream_token
+
+        db_name = self.gen_unique_name("test_mdb_cross_db")
+        c_name_1 = self.gen_unique_name("test_mdb_cross_col1")
+        c_name_2 = self.gen_unique_name("test_mdb_cross_col2")
+
+        self.resources_to_cleanup.append(("collection_in_db", (db_name, c_name_1)))
+        self.resources_to_cleanup.append(("collection_in_db", (db_name, c_name_2)))
+        self.resources_to_cleanup.append(("database", db_name))
+
+        # Initial cleanup
+        self.cleanup_database(upstream_client, db_name)
+
+        # Step 1: Create database
+        upstream_client.create_database(db_name)
+
+        # Step 2: Create DB-scoped client and 1st collection with insert
+        up_db_client = MilvusClient(uri=upstream_uri, token=upstream_token, db_name=db_name)
+        schema1 = self.create_default_schema(up_db_client)
+        up_db_client.create_collection(collection_name=c_name_1, schema=schema1, consistency_level="Strong")
+        up_db_client.insert(c_name_1, self.generate_test_data(100))
+        up_db_client.flush(c_name_1)
+
+        # Step 3: Alter DB properties
+        upstream_client.alter_database_properties(
+            db_name=db_name,
+            properties={"database.max.collections": 10},
+        )
+
+        # Step 4: Create 2nd collection in the same DB
+        schema2 = self.create_default_schema(up_db_client)
+        up_db_client.create_collection(collection_name=c_name_2, schema=schema2, consistency_level="Strong")
+        up_db_client.close()
+
+        # Wait for all operations to sync to downstream
+
+        # Check database exists on downstream
+        def check_db_exists():
+            return db_name in downstream_client.list_databases()
+
+        assert self.wait_for_sync(check_db_exists, sync_timeout, f"create database {db_name}")
+
+        # Check 1st collection exists on downstream
+        def check_col1():
+            try:
+                dn = MilvusClient(uri=downstream_uri, token=downstream_token, db_name=db_name)
+                exists = dn.has_collection(c_name_1)
+                dn.close()
+                return exists
+            except Exception as e:
+                logger.warning(f"Check col1 sync failed: {e}")
+                return False
+
+        assert self.wait_for_sync(check_col1, sync_timeout, f"collection {c_name_1} in {db_name}")
+
+        # Check 2nd collection exists on downstream
+        def check_col2():
+            try:
+                dn = MilvusClient(uri=downstream_uri, token=downstream_token, db_name=db_name)
+                exists = dn.has_collection(c_name_2)
+                dn.close()
+                return exists
+            except Exception as e:
+                logger.warning(f"Check col2 sync failed: {e}")
+                return False
+
+        assert self.wait_for_sync(check_col2, sync_timeout, f"collection {c_name_2} in {db_name}")
+
+        # Check DB properties synced
+        def check_db_props():
+            try:
+                if db_name not in downstream_client.list_databases():
+                    return False
+                props = downstream_client.describe_database(db_name)
+                logger.info(f"Downstream database properties: {props}")
+                return str(props.get("database.max.collections", "")) == "10"
+            except Exception as e:
+                logger.warning(f"Check DB properties sync failed: {e}")
+                return False
+
+        assert self.wait_for_sync(check_db_props, sync_timeout, f"DB properties sync for {db_name}")

tests/python_client/cdc/testcases/test_partition.py

@@ -0,0 +1,513 @@
+"""
+CDC sync tests for partition operations.
+"""
+
+import time
+import pytest
+from common.common_type import CaseLabel
+from .base import TestCDCSyncBase
+
+
+@pytest.mark.tags(CaseLabel.CDC)
+class TestCDCSyncPartition(TestCDCSyncBase):
+    """Test CDC sync for partition operations."""
+
+    def setup_method(self):
+        """Setup for each test method."""
+        self.resources_to_cleanup = []
+
+    def teardown_method(self):
+        """Cleanup after each test method - only cleanup upstream, downstream will sync."""
+        upstream_client = getattr(self, "_upstream_client", None)
+
+        if upstream_client:
+            for resource_type, resource_name in self.resources_to_cleanup:
+                if resource_type == "collection":
+                    self.cleanup_collection(upstream_client, resource_name)
+
+            time.sleep(1)  # Allow cleanup to sync to downstream
+
+    def test_create_partition(self, upstream_client, downstream_client, sync_timeout):
+        """Test CREATE_PARTITION operation sync."""
+        # Store upstream client for teardown
+        self._upstream_client = upstream_client
+
+        collection_name = self.gen_unique_name("test_col_part_create")
+        partition_name = self.gen_unique_name("test_part_create")
+        self.resources_to_cleanup.append(("collection", collection_name))
+
+        # Initial cleanup
+        self.cleanup_collection(upstream_client, collection_name)
+
+        # Create collection
+        upstream_client.create_collection(
+            collection_name=collection_name,
+            schema=self.create_default_schema(upstream_client),
+        )
+
+        # Wait for creation to sync
+        def check_create():
+            return downstream_client.has_collection(collection_name)
+
+        assert self.wait_for_sync(
+            check_create, sync_timeout, f"create collection {collection_name}"
+        )
+
+        # Create partition
+        upstream_client.create_partition(collection_name, partition_name)
+
+        # Verify partition exists in upstream
+        upstream_partitions = upstream_client.list_partitions(collection_name)
+        assert partition_name in upstream_partitions
+
+        # Wait for partition sync to downstream
+        def check_partition():
+            try:
+                downstream_partitions = downstream_client.list_partitions(
+                    collection_name
+                )
+                return partition_name in downstream_partitions
+            except:
+                return False
+
+        assert self.wait_for_sync(
+            check_partition, sync_timeout, f"create partition {partition_name}"
+        )
+
+    def test_drop_partition(self, upstream_client, downstream_client, sync_timeout):
+        """Test DROP_PARTITION operation sync."""
+        # Store upstream client for teardown
+        self._upstream_client = upstream_client
+
+        collection_name = self.gen_unique_name("test_col_part_drop")
+        partition_name = self.gen_unique_name("test_part_drop")
+        self.resources_to_cleanup.append(("collection", collection_name))
+
+        # Initial cleanup
+        self.cleanup_collection(upstream_client, collection_name)
+
+        # Create collection and partition
+        upstream_client.create_collection(
+            collection_name=collection_name,
+            schema=self.create_default_schema(upstream_client),
+        )
+        upstream_client.create_partition(collection_name, partition_name)
+
+        # Wait for setup to sync
+        def check_setup():
+            try:
+                return downstream_client.has_collection(
+                    collection_name
+                ) and partition_name in downstream_client.list_partitions(
+                    collection_name
+                )
+            except:
+                return False
+
+        assert self.wait_for_sync(
+            check_setup,
+            sync_timeout,
+            f"setup collection and partition {collection_name}",
+        )
+
+        # Drop partition
+        upstream_client.drop_partition(collection_name, partition_name)
+
+        # Verify partition is dropped in upstream
+        upstream_partitions = upstream_client.list_partitions(collection_name)
+        assert partition_name not in upstream_partitions
+
+        # Wait for drop to sync to downstream
+        def check_drop():
+            try:
+                downstream_partitions = downstream_client.list_partitions(
+                    collection_name
+                )
+                return partition_name not in downstream_partitions
+            except:
+                return True  # If error, assume partition is dropped
+
+        assert self.wait_for_sync(
+            check_drop, sync_timeout, f"drop partition {partition_name}"
+        )
+
+    def test_load_partition(self, upstream_client, downstream_client, sync_timeout):
+        """Test LOAD_PARTITION operation sync."""
+        # Store upstream client for teardown
+        self._upstream_client = upstream_client
+
+        collection_name = self.gen_unique_name("test_col_part_load")
+        partition_name = self.gen_unique_name("test_part_load")
+        self.resources_to_cleanup.append(("collection", collection_name))
+
+        # Initial cleanup
+        self.cleanup_collection(upstream_client, collection_name)
+
+        # Create collection, partition, and index
+        upstream_client.create_collection(
+            collection_name=collection_name,
+            schema=self.create_default_schema(upstream_client),
+        )
+        upstream_client.create_partition(collection_name, partition_name)
+
+        # Create index and load collection (required for querying/searching)
+        index_params = upstream_client.prepare_index_params()
+        index_params.add_index(
+            field_name="vector", index_type="AUTOINDEX", metric_type="L2"
+        )
+        upstream_client.create_index(collection_name, index_params)
+
+        # Wait for setup to sync
+        def check_setup():
+            try:
+                return downstream_client.has_collection(
+                    collection_name
+                ) and partition_name in downstream_client.list_partitions(
+                    collection_name
+                )
+            except:
+                return False
+
+        assert self.wait_for_sync(
+            check_setup,
+            sync_timeout,
+            f"setup collection and partition {collection_name}",
+        )
+
+        # Load partition
+        upstream_client.load_partitions(collection_name, [partition_name])
+        # check partition load state in upstream
+        upstream_load_state = upstream_client.get_load_state(
+            collection_name, partition_name
+        )
+        load_state = str(upstream_load_state["state"])
+        print(f"DEBUG: partition load state in upstream: {load_state}")
+
+        # Wait for load to sync
+        def check_load():
+            try:
+                # Check partition load state
+                load_state = downstream_client.get_load_state(
+                    collection_name=collection_name, partition_name=partition_name
+                )
+                print(
+                    f"DEBUG: partition load state in check_load: {load_state['state']}"
+                )
+                return "Loaded" == str(load_state["state"])
+            except Exception as e:
+                print(f"DEBUG: get_load_state exception in check_load: {e}")
+                return False
+
+        assert self.wait_for_sync(
+            check_load, sync_timeout, f"load partition {partition_name}"
+        )
+
+    def test_release_partition(self, upstream_client, downstream_client, sync_timeout):
+        """Test RELEASE_PARTITION operation sync."""
+        # Store upstream client for teardown
+        self._upstream_client = upstream_client
+
+        collection_name = self.gen_unique_name("test_col_part_release")
+        partition_name = self.gen_unique_name("test_part_release")
+        self.resources_to_cleanup.append(("collection", collection_name))
+
+        # Initial cleanup
+        self.cleanup_collection(upstream_client, collection_name)
+
+        # Create collection, partition, index, and load
+        upstream_client.create_collection(
+            collection_name=collection_name,
+            schema=self.create_default_schema(upstream_client),
+        )
+        upstream_client.create_partition(collection_name, partition_name)
+
+        # Create index and load collection (required for querying/searching)
+        index_params = upstream_client.prepare_index_params()
+        index_params.add_index(
+            field_name="vector", index_type="AUTOINDEX", metric_type="L2"
+        )
+        upstream_client.create_index(collection_name, index_params)
+        upstream_client.load_collection(collection_name)
+        upstream_client.load_partitions(collection_name, [partition_name])
+        for p_name in [partition_name, None]:
+            data = [{"vector": [0.1] * 128, "id": i} for i in range(100)]
+            upstream_client.insert(collection_name, data, partition_name=p_name)
+
+        # Wait for setup to sync
+        def check_setup():
+            try:
+                query_vector = [[0.1] * 128]
+                downstream_client.search(
+                    collection_name=collection_name,
+                    data=query_vector,
+                    limit=1,
+                    partition_names=[partition_name],
+                    output_fields=[],
+                )
+                return True
+            except:
+                return False
+
+        assert self.wait_for_sync(
+            check_setup, sync_timeout, f"setup and load partition {partition_name}"
+        )
+
+        # Release partition
+        upstream_client.release_partitions(collection_name, [partition_name])
+
+        # check partition load state in upstream
+        upstream_load_state = upstream_client.get_load_state(
+            collection_name, partition_name
+        )
+        load_state = str(upstream_load_state["state"])
+        print(f"DEBUG: partition load state in upstream: {load_state}")
+
+        # check partition load state in upstream
+        def check_release():
+            try:
+                query_vector = [[0.1] * 128]
+                res = upstream_client.search(
+                    collection_name=collection_name,
+                    data=query_vector,
+                    limit=1,
+                    partition_names=[partition_name],
+                    output_fields=[],
+                )
+                print(
+                    f"DEBUG: released partition {partition_name} can still be searched: {res}"
+                )
+                print(
+                    f"DEBUG: released partition {partition_name} can still be searched"
+                )
+                return False
+            except:
+                print(f"DEBUG: released partition {partition_name} cannot be searched")
+                return True
+
+        assert self.wait_for_sync(
+            check_release, sync_timeout, f"release partition {partition_name}"
+        )
+
+        # check partition load state in downstream
+        def check_release():
+            try:
+                query_vector = [[0.1] * 128]
+                downstream_client.search(
+                    collection_name=collection_name,
+                    data=query_vector,
+                    limit=1,
+                    partition_names=[partition_name],
+                    output_fields=[],
+                )
+                print(
+                    f"DEBUG: released partition {partition_name} can still be searched"
+                )
+                return False
+            except:
+                print(f"DEBUG: released partition {partition_name} cannot be searched")
+                return True
+
+        assert self.wait_for_sync(
+            check_release, sync_timeout, f"release partition {partition_name}"
+        )
+
+    def test_partition_insert(self, upstream_client, downstream_client, sync_timeout):
+        """Test INSERT operation to partition sync."""
+        # Store upstream client for teardown
+        self._upstream_client = upstream_client
+
+        collection_name = self.gen_unique_name("test_col_part_insert")
+        partition_name = self.gen_unique_name("test_part_insert")
+        self.resources_to_cleanup.append(("collection", collection_name))
+
+        # Initial cleanup
+        self.cleanup_collection(upstream_client, collection_name)
+
+        # Create collection and partition
+        upstream_client.create_collection(
+            collection_name=collection_name,
+            schema=self.create_default_schema(upstream_client),
+        )
+        upstream_client.create_partition(collection_name, partition_name)
+
+        # Create index and load collection (required for querying/searching)
+        index_params = upstream_client.prepare_index_params()
+        index_params.add_index(
+            field_name="vector", index_type="AUTOINDEX", metric_type="L2"
+        )
+        upstream_client.create_index(collection_name, index_params)
+        upstream_client.load_collection(collection_name)
+
+        # Wait for setup to sync
+        def check_setup():
+            try:
+                return downstream_client.has_collection(
+                    collection_name
+                ) and partition_name in downstream_client.list_partitions(
+                    collection_name
+                )
+            except:
+                return False
+
+        assert self.wait_for_sync(
+            check_setup,
+            sync_timeout,
+            f"setup collection and partition {collection_name}",
+        )
+
+        # Insert data to specific partition
+        test_data = self.generate_test_data(100)
+        result = upstream_client.insert(
+            collection_name, test_data, partition_name=partition_name
+        )
+        inserted_count = result.get("insert_count", len(test_data))
+
+        # Flush to ensure data is persisted
+        upstream_client.flush(collection_name)
+
+        # Wait for data sync to downstream partition by querying
+        def check_data():
+            try:
+                # Query data in specific partition
+                result = downstream_client.query(
+                    collection_name=collection_name,
+                    filter="",
+                    output_fields=["count(*)"],
+                    partition_names=[partition_name],
+                )
+                count = result[0]["count(*)"] if result else 0
+                return count >= inserted_count
+            except:
+                return False
+
+        assert self.wait_for_sync(
+            check_data, sync_timeout, f"insert data to partition {partition_name}"
+        )
+
+    def test_partition_delete(self, upstream_client, downstream_client, sync_timeout):
+        """Test DELETE operation from partition sync."""
+        # Store upstream client for teardown
+        self._upstream_client = upstream_client
+
+        collection_name = self.gen_unique_name("test_col_part_delete")
+        partition_name = self.gen_unique_name("test_part_delete")
+        self.resources_to_cleanup.append(("collection", collection_name))
+
+        # Initial cleanup
+        self.cleanup_collection(upstream_client, collection_name)
+
+        # Create collection and partition
+        upstream_client.create_collection(
+            collection_name=collection_name,
+            schema=self.create_default_schema(upstream_client),
+            consistency_level="Strong",
+        )
+        upstream_client.create_partition(collection_name, partition_name)
+
+        # Create index and load collection (required for querying/searching)
+        index_params = upstream_client.prepare_index_params()
+        index_params.add_index(
+            field_name="vector", index_type="AUTOINDEX", metric_type="L2"
+        )
+        upstream_client.create_index(collection_name, index_params)
+        upstream_client.load_collection(collection_name)
+
+        # Wait for setup to sync
+        def check_setup():
+            try:
+                return downstream_client.has_collection(
+                    collection_name
+                ) and partition_name in downstream_client.list_partitions(
+                    collection_name
+                )
+            except:
+                return False
+
+        assert self.wait_for_sync(
+            check_setup,
+            sync_timeout,
+            f"setup collection and partition {collection_name}",
+        )
+
+        # Insert data to partition
+        test_data = self.generate_test_data(100)
+        upstream_client.insert(
+            collection_name, test_data, partition_name=partition_name
+        )
+        upstream_client.flush(collection_name)
+
+        # Wait for initial data sync by querying partition
+        def check_data():
+            try:
+                result = downstream_client.query(
+                    collection_name=collection_name,
+                    filter="",
+                    output_fields=["count(*)"],
+                    partition_names=[partition_name],
+                )
+                count = result[0]["count(*)"] if result else 0
+                return count >= 100
+            except:
+                return False
+
+        assert self.wait_for_sync(
+            check_data, sync_timeout, f"initial data sync to partition {partition_name}"
+        )
+
+        # Delete some data from partition
+        delete_ids = list(range(20))  # Delete first 20 records
+        upstream_client.delete(
+            collection_name, filter=f"id in {delete_ids}", partition_name=partition_name
+        )
+        upstream_client.flush(collection_name)
+        deleted_result = upstream_client.query(
+            collection_name=collection_name,
+            filter=f"id in {delete_ids}",
+            output_fields=["id"],
+            partition_names=[partition_name],
+        )
+        total_count = upstream_client.query(
+            collection_name=collection_name,
+            filter="",
+            output_fields=["count(*)"],
+            partition_names=[partition_name],
+        )
+        total_count = total_count[0]["count(*)"] if total_count else 0
+        print(f"DEBUG: deleted_result in upstream: {deleted_result}")
+        print(f"DEBUG: total_count in upstream: {total_count}")
+
+        # Wait for delete to sync by querying partition
+        def check_delete():
+            try:
+                # Query for the deleted records in partition - should return empty
+                deleted_result = downstream_client.query(
+                    collection_name=collection_name,
+                    filter=f"id in {delete_ids}",
+                    output_fields=["id"],
+                    partition_names=[partition_name],
+                )
+                # Query total count in partition
+                count_result = downstream_client.query(
+                    collection_name=collection_name,
+                    filter="",
+                    output_fields=["count(*)"],
+                    partition_names=[partition_name],
+                )
+                deleted_count = len(deleted_result) if deleted_result else 0
+                total_count = count_result[0]["count(*)"] if count_result else 0
+
+                print(f"DEBUG: deleted_result in check_delete: {deleted_result}")
+                print(f"DEBUG: count_result in check_delete: {count_result}")
+                print(
+                    f"DEBUG: deleted_count: {deleted_count}, total_count: {total_count}"
+                )
+
+                # Verify deleted records are gone and total count is correct in partition
+                return deleted_count == 0 and total_count == 80
+            except Exception as e:
+                print(f"DEBUG: query exception in check_delete: {e}")
+                return False
+
+        assert self.wait_for_sync(
+            check_delete, sync_timeout, f"delete data from partition {partition_name}"
+        )

tests/python_client/cdc/testcases/test_rbac.py

@@ -0,0 +1,873 @@
+"""
+CDC sync tests for RBAC operations.
+"""
+
+import time
+import pytest
+from common.common_type import CaseLabel
+from .base import TestCDCSyncBase, logger
+
+
+@pytest.mark.tags(CaseLabel.CDC)
+class TestCDCSyncRBAC(TestCDCSyncBase):
+    """Test CDC sync for RBAC operations."""
+
+    def setup_method(self):
+        """Setup for each test method."""
+        self.resources_to_cleanup = []
+
+    def teardown_method(self):
+        """Cleanup after each test method - only cleanup upstream, downstream will sync."""
+        upstream_client = getattr(self, "_upstream_client", None)
+
+        if upstream_client:
+            for resource_type, resource_name in self.resources_to_cleanup:
+                if resource_type == "user":
+                    self.cleanup_user(upstream_client, resource_name)
+                elif resource_type == "role":
+                    self.cleanup_role(upstream_client, resource_name)
+
+            time.sleep(1)  # Allow cleanup to sync to downstream
+
+    def test_create_role(self, upstream_client, downstream_client, sync_timeout):
+        """Test CREATE_ROLE operation sync."""
+        # Store upstream client for teardown
+        self._upstream_client = upstream_client
+
+        role_name = self.gen_unique_name("test_role_create")
+        self.resources_to_cleanup.append(("role", role_name))
+
+        # Initial cleanup
+        self.cleanup_role(upstream_client, role_name)
+
+        # Create role in upstream
+        upstream_client.create_role(role_name)
+        assert role_name in upstream_client.list_roles()
+
+        # Wait for sync to downstream
+        def check_sync():
+            return role_name in downstream_client.list_roles()
+
+        assert self.wait_for_sync(check_sync, sync_timeout, f"create role {role_name}")
+
+    def test_drop_role(self, upstream_client, downstream_client, sync_timeout):
+        """Test DROP_ROLE operation sync."""
+        # Store upstream client for teardown
+        self._upstream_client = upstream_client
+
+        role_name = self.gen_unique_name("test_role_drop")
+        self.resources_to_cleanup.append(("role", role_name))
+
+        # Initial cleanup
+        self.cleanup_role(upstream_client, role_name)
+
+        # Create role first
+        upstream_client.create_role(role_name)
+
+        # Wait for creation to sync
+        def check_create():
+            return role_name in downstream_client.list_roles()
+
+        assert self.wait_for_sync(
+            check_create, sync_timeout, f"create role {role_name}"
+        )
+
+        # Drop role in upstream
+        upstream_client.drop_role(role_name)
+        assert role_name not in upstream_client.list_roles()
+
+        # Wait for drop to sync
+        def check_drop():
+            return role_name not in downstream_client.list_roles()
+
+        assert self.wait_for_sync(check_drop, sync_timeout, f"drop role {role_name}")
+
+    def test_create_user(self, upstream_client, downstream_client, sync_timeout):
+        """Test CREATE_USER operation sync."""
+        # Store upstream client for teardown
+        self._upstream_client = upstream_client
+
+        username = self.gen_unique_name("test_user_create", max_length=31)
+        password = "TestPass123!"
+        self.resources_to_cleanup.append(("user", username))
+
+        # Initial cleanup
+        self.cleanup_user(upstream_client, username)
+
+        # Create user in upstream
+        upstream_client.create_user(username, password)
+        assert username in upstream_client.list_users()
+
+        # Wait for sync to downstream
+        def check_sync():
+            return username in downstream_client.list_users()
+
+        assert self.wait_for_sync(check_sync, sync_timeout, f"create user {username}")
+
+    def test_drop_user(self, upstream_client, downstream_client, sync_timeout):
+        """Test DROP_USER operation sync."""
+        # Store upstream client for teardown
+        self._upstream_client = upstream_client
+
+        username = self.gen_unique_name("test_user_drop", max_length=31)
+        password = "TestPass123!"
+        self.resources_to_cleanup.append(("user", username))
+
+        # Initial cleanup
+        self.cleanup_user(upstream_client, username)
+
+        # Create user first
+        upstream_client.create_user(username, password)
+
+        # Wait for creation to sync
+        def check_create():
+            return username in downstream_client.list_users()
+
+        assert self.wait_for_sync(check_create, sync_timeout, f"create user {username}")
+
+        # Drop user in upstream
+        upstream_client.drop_user(username)
+        assert username not in upstream_client.list_users()
+
+        # Wait for drop to sync
+        def check_drop():
+            return username not in downstream_client.list_users()
+
+        assert self.wait_for_sync(check_drop, sync_timeout, f"drop user {username}")
+
+    def test_grant_role(self, upstream_client, downstream_client, sync_timeout):
+        """Test GRANT_ROLE operation sync."""
+        # Store upstream client for teardown
+        self._upstream_client = upstream_client
+
+        username = self.gen_unique_name("test_user_grant", max_length=31)
+        role_name = self.gen_unique_name("test_role_grant")
+        password = "TestPass123!"
+        self.resources_to_cleanup.append(("user", username))
+        self.resources_to_cleanup.append(("role", role_name))
+
+        # Initial cleanup
+        self.cleanup_user(upstream_client, username)
+        self.cleanup_role(upstream_client, role_name)
+
+        # Create user and role
+        upstream_client.create_user(username, password)
+        upstream_client.create_role(role_name)
+
+        # Wait for creation to sync
+        def check_create():
+            return (
+                username in downstream_client.list_users()
+                and role_name in downstream_client.list_roles()
+            )
+
+        assert self.wait_for_sync(
+            check_create, sync_timeout, "create user/role for grant"
+        )
+
+        # Grant role to user
+        upstream_client.grant_role(username, role_name)
+
+        # Wait for grant to sync
+        def check_grant():
+            # Allow operation to propagate
+            time.sleep(2)
+            # Verify role is bound to user using describe_user
+            try:
+                user_info = downstream_client.describe_user(username)
+                user_roles = user_info.get("roles", [])
+                print(f"DEBUG: user_roles in check_grant: {user_roles}")
+                return role_name in user_roles
+            except Exception as e:
+                logger.debug(f"Error checking user roles: {e}")
+                return False
+
+        assert self.wait_for_sync(
+            check_grant, sync_timeout, f"grant role {role_name} to user {username}"
+        )
+
+    def test_revoke_role(self, upstream_client, downstream_client, sync_timeout):
+        """Test REVOKE_ROLE operation sync."""
+        # Store upstream client for teardown
+        self._upstream_client = upstream_client
+
+        username = self.gen_unique_name("test_user_revoke", max_length=31)
+        role_name = self.gen_unique_name("test_role_revoke")
+        password = "TestPass123!"
+        self.resources_to_cleanup.append(("user", username))
+        self.resources_to_cleanup.append(("role", role_name))
+
+        # Initial cleanup
+        self.cleanup_user(upstream_client, username)
+        self.cleanup_role(upstream_client, role_name)
+
+        # Create user and role, then grant role
+        upstream_client.create_user(username, password)
+        upstream_client.create_role(role_name)
+        upstream_client.grant_role(username, role_name)
+
+        # Wait for setup to sync
+        time.sleep(5)
+
+        # Revoke role from user
+        upstream_client.revoke_role(username, role_name)
+
+        # Wait for revoke to sync
+        def check_revoke():
+            time.sleep(2)  # Allow operation to propagate
+            # Verify role is removed from user using describe_user
+            try:
+                user_info = downstream_client.describe_user(username)
+                user_roles = user_info.get("roles", [])
+                print(f"DEBUG: user_roles in check_revoke: {user_roles}")
+                return role_name not in user_roles
+            except Exception as e:
+                logger.debug(f"Error checking user roles: {e}")
+                return False
+
+        assert self.wait_for_sync(
+            check_revoke, sync_timeout, f"revoke role {role_name} from user {username}"
+        )
+
+    def test_grant_privilege(self, upstream_client, downstream_client, sync_timeout):
+        """Test GRANT_PRIVILEGE operation sync."""
+        # Store upstream client for teardown
+        self._upstream_client = upstream_client
+
+        role_name = self.gen_unique_name("test_role_priv_grant")
+        self.resources_to_cleanup.append(("role", role_name))
+
+        # Initial cleanup
+        self.cleanup_role(upstream_client, role_name)
+
+        # Create role
+        upstream_client.create_role(role_name)
+
+        # Wait for creation to sync
+        def check_create():
+            return role_name in downstream_client.list_roles()
+
+        assert self.wait_for_sync(
+            check_create, sync_timeout, f"create role for privilege {role_name}"
+        )
+
+        # Grant privilege to role
+        upstream_client.grant_privilege(
+            role_name=role_name,
+            object_type="Collection",
+            privilege="Search",
+            object_name="*",
+        )
+
+        # check privilege in upstream
+        upstream_privilege = upstream_client.describe_role(role_name)
+        print(f"DEBUG: upstream_privilege in check_grant: {upstream_privilege}")
+
+        # Wait for privilege grant to sync
+        def check_grant():
+            time.sleep(2)
+            # Verify privilege is actually granted using describe_role
+            try:
+                role_info = downstream_client.describe_role(role_name)
+                print(f"DEBUG: role_info in check_grant: {role_info}")
+                for privilege_info in role_info["privileges"]:
+                    print(f"DEBUG: privilege_info in check_grant: {privilege_info}")
+                    print(
+                        f"DEBUG: privilege_info.get('object_type') in check_grant: {privilege_info.get('object_type')}"
+                    )
+                    print(
+                        f"DEBUG: privilege_info.get('privilege') in check_grant: {privilege_info.get('privilege')}"
+                    )
+                    print(
+                        f"DEBUG: privilege_info.get('object_name') in check_grant: {privilege_info.get('object_name')}"
+                    )
+                    if (
+                        privilege_info.get("object_type") == "Collection"
+                        and privilege_info.get("privilege") == "Search"
+                        and privilege_info.get("object_name") == "*"
+                    ):
+                        return True
+                return False
+            except Exception as e:
+                logger.debug(f"Error checking role privileges: {e}")
+                return False
+
+        assert self.wait_for_sync(
+            check_grant, sync_timeout, f"grant privilege to role {role_name}"
+        )
+
+    def test_revoke_privilege(self, upstream_client, downstream_client, sync_timeout):
+        """Test REVOKE_PRIVILEGE operation sync."""
+        # Store upstream client for teardown
+        self._upstream_client = upstream_client
+
+        role_name = self.gen_unique_name("test_role_priv_revoke")
+        self.resources_to_cleanup.append(("role", role_name))
+
+        # Initial cleanup
+        self.cleanup_role(upstream_client, role_name)
+
+        # Create role and grant privilege
+        upstream_client.create_role(role_name)
+
+        upstream_client.grant_privilege(
+            role_name=role_name,
+            object_type="Collection",
+            privilege="Search",
+            object_name="*",
+        )
+
+        # Wait for setup
+        time.sleep(3)
+
+        # Revoke privilege from role
+        upstream_client.revoke_privilege(
+            role_name=role_name,
+            object_type="Collection",
+            privilege="Search",
+            object_name="*",
+        )
+
+        # Wait for revoke to sync
+        def check_revoke():
+            time.sleep(2)
+            # Verify privilege is actually revoked using describe_role
+            try:
+                role_info = downstream_client.describe_role(role_name)
+                for privilege_info in role_info["privileges"]:
+                    if (
+                        privilege_info.get("object_type") == "Collection"
+                        and privilege_info.get("privilege") == "Search"
+                        and privilege_info.get("object_name") == "*"
+                    ):
+                        return False  # Should not find the privilege
+                return True
+            except Exception as e:
+                logger.debug(f"Error checking role privileges: {e}")
+                return False
+
+        assert self.wait_for_sync(
+            check_revoke, sync_timeout, f"revoke privilege from role {role_name}"
+        )
+
+    def test_update_password(
+        self, upstream_client, downstream_client, sync_timeout, downstream_uri
+    ):
+        """Test UPDATE_PASSWORD operation sync."""
+        # Store upstream client for teardown
+        self._upstream_client = upstream_client
+
+        username = self.gen_unique_name("test_user_pwd", max_length=31)
+        old_password = "OldPass123!"
+        new_password = "NewPass456!"
+        self.resources_to_cleanup.append(("user", username))
+
+        # Initial cleanup
+        self.cleanup_user(upstream_client, username)
+
+        # Create user first
+        upstream_client.create_user(username, old_password)
+
+        # Wait for creation to sync
+        def check_create():
+            return username in downstream_client.list_users()
+
+        assert self.wait_for_sync(check_create, sync_timeout, f"create user {username}")
+
+        # Update password in upstream
+        upstream_client.update_password(username, old_password, new_password)
+
+        # Wait for password update to sync
+        def check_password_update():
+            time.sleep(2)
+            # Verify password update by trying to create a client with new credentials
+            try:
+                from pymilvus import MilvusClient
+
+                # Try to connect with new password
+                test_client = MilvusClient(
+                    uri=downstream_uri, token=f"{username}:{new_password}"
+                )
+                # Simple operation to verify connection works
+                test_client.list_collections()
+                test_client.close()
+                return True
+            except Exception as e:
+                logger.debug(f"Error verifying new password: {e}")
+                return False
+
+        assert self.wait_for_sync(
+            check_password_update, sync_timeout, f"update password for user {username}"
+        )
+
+    def test_create_privilege_group(
+        self, upstream_client, downstream_client, sync_timeout
+    ):
+        """Test CREATE_PRIVILEGE_GROUP operation sync."""
+        # Store upstream client for teardown
+        self._upstream_client = upstream_client
+
+        group_name = self.gen_unique_name("test_priv_group")
+        # Note: privilege groups need special cleanup handling
+
+        # Initial cleanup - try to drop if exists
+        try:
+            upstream_client.drop_privilege_group(group_name)
+        except:
+            pass  # Ignore if doesn't exist
+
+        # Create privilege group in upstream
+        upstream_client.create_privilege_group(group_name)
+
+        # Verify in upstream
+        def check_create():
+            try:
+                upstream_groups = upstream_client.list_privilege_groups()
+                print(f"DEBUG: upstream_groups in check_create: {upstream_groups}")
+                return group_name in [
+                    group["privilege_group"] for group in upstream_groups
+                ]
+            except Exception as e:
+                logger.debug(f"Error checking privilege groups: {e}")
+                return False
+
+        assert self.wait_for_sync(
+            check_create, sync_timeout, f"create privilege group {group_name}"
+        )
+
+        # Wait for sync to downstream
+        def check_sync():
+            try:
+                downstream_groups = downstream_client.list_privilege_groups()
+                return group_name in [
+                    group["privilege_group"] for group in downstream_groups
+                ]
+            except Exception as e:
+                logger.debug(f"Error checking privilege groups: {e}")
+                return False
+
+        assert self.wait_for_sync(
+            check_sync, sync_timeout, f"create privilege group {group_name}"
+        )
+
+        # Cleanup
+        try:
+            upstream_client.drop_privilege_group(group_name)
+        except:
+            pass
+
+    def test_drop_privilege_group(
+        self, upstream_client, downstream_client, sync_timeout
+    ):
+        """Test DROP_PRIVILEGE_GROUP operation sync."""
+        # Store upstream client for teardown
+        self._upstream_client = upstream_client
+
+        group_name = self.gen_unique_name("test_priv_group_drop")
+
+        # Initial cleanup
+        try:
+            upstream_client.drop_privilege_group(group_name)
+        except:
+            pass
+
+        # Create privilege group first
+        upstream_client.create_privilege_group(group_name)
+
+        # Wait for creation to sync
+        def check_create():
+            try:
+                return group_name in [
+                    group["privilege_group"]
+                    for group in downstream_client.list_privilege_groups()
+                ]
+            except Exception as e:
+                logger.debug(f"Error checking privilege groups: {e}")
+                return False
+
+        assert self.wait_for_sync(
+            check_create, sync_timeout, f"create privilege group {group_name}"
+        )
+
+        # Drop privilege group in upstream
+        upstream_client.drop_privilege_group(group_name)
+
+        # Verify dropped in upstream
+        upstream_groups = upstream_client.list_privilege_groups()
+        assert group_name not in [group["privilege_group"] for group in upstream_groups]
+
+        # Wait for drop to sync
+        def check_drop():
+            try:
+                downstream_groups = downstream_client.list_privilege_groups()
+                return group_name not in [
+                    group["privilege_group"] for group in downstream_groups
+                ]
+            except Exception as e:
+                logger.debug(f"Error checking privilege groups: {e}")
+                return False
+
+        assert self.wait_for_sync(
+            check_drop, sync_timeout, f"drop privilege group {group_name}"
+        )
+
+    def test_grant_privilege_v2(self, upstream_client, downstream_client, sync_timeout):
+        """Test GRANT_PRIVILEGE_V2 operation sync."""
+        # Store upstream client for teardown
+        self._upstream_client = upstream_client
+
+        role_name = self.gen_unique_name("test_role_priv_v2")
+        self.resources_to_cleanup.append(("role", role_name))
+
+        # Initial cleanup
+        self.cleanup_role(upstream_client, role_name)
+
+        # Create role
+        upstream_client.create_role(role_name)
+
+        # Wait for creation to sync
+        def check_create():
+            return role_name in downstream_client.list_roles()
+
+        assert self.wait_for_sync(
+            check_create, sync_timeout, f"create role for privilege v2 {role_name}"
+        )
+
+        # Grant privilege using v2 API
+        upstream_client.grant_privilege_v2(
+            role_name=role_name, privilege="Query", collection_name="*"
+        )
+        # check privilege in upstream
+        upstream_privilege = upstream_client.describe_role(role_name)["privileges"]
+        print(f"DEBUG: upstream_privilege in check_grant: {upstream_privilege}")
+
+        # Wait for privilege grant to sync
+        def check_grant():
+            time.sleep(2)
+            # Verify privilege is actually granted using describe_role
+            try:
+                role_info = downstream_client.describe_role(role_name)
+                for privilege_info in role_info["privileges"]:
+                    if (
+                        privilege_info.get("privilege") == "Query"
+                        and privilege_info.get("object_name") == "*"
+                    ):
+                        return True
+                return False
+            except Exception as e:
+                logger.debug(f"Error checking role privileges v2: {e}")
+                return False
+
+        assert self.wait_for_sync(
+            check_grant, sync_timeout, f"grant privilege v2 to role {role_name}"
+        )
+
+    def test_revoke_privilege_v2(
+        self, upstream_client, downstream_client, sync_timeout
+    ):
+        """Test REVOKE_PRIVILEGE_V2 operation sync."""
+        # Store upstream client for teardown
+        self._upstream_client = upstream_client
+
+        role_name = self.gen_unique_name("test_role_revoke_v2")
+        self.resources_to_cleanup.append(("role", role_name))
+
+        # Initial cleanup
+        self.cleanup_role(upstream_client, role_name)
+
+        # Create role and grant privilege using v2 API
+        upstream_client.create_role(role_name)
+        upstream_client.grant_privilege_v2(
+            role_name=role_name, privilege="Query", collection_name="*"
+        )
+
+        # Wait for setup
+        time.sleep(3)
+
+        # check privilege in upstream
+        upstream_privilege = upstream_client.describe_role(role_name)["privileges"]
+        print(f"DEBUG: upstream_privilege in check_revoke: {upstream_privilege}")
+
+        # check privilege in downstream
+        def check_grant():
+            try:
+                role_info = downstream_client.describe_role(role_name)
+                for privilege_info in role_info["privileges"]:
+                    if (
+                        privilege_info.get("privilege") == "Query"
+                        and privilege_info.get("object_name") == "*"
+                    ):
+                        return True
+                return False
+            except Exception as e:
+                logger.debug(f"Error checking role privileges v2: {e}")
+                return False
+
+        assert self.wait_for_sync(
+            check_grant, sync_timeout, f"grant privilege v2 to role {role_name}"
+        )
+
+        # Revoke privilege using v2 API
+        upstream_client.revoke_privilege_v2(
+            role_name=role_name, privilege="Query", collection_name="*"
+        )
+        # check privilege in upstream
+        upstream_privilege = upstream_client.describe_role(role_name)["privileges"]
+        print(f"DEBUG: upstream_privilege in check_revoke: {upstream_privilege}")
+
+        # Wait for revoke to sync
+        def check_revoke():
+            time.sleep(2)
+            # Verify privilege is actually revoked using describe_role
+            try:
+                role_info = downstream_client.describe_role(role_name)
+                for privilege_info in role_info["privileges"]:
+                    if (
+                        privilege_info.get("privilege") == "Query"
+                        and privilege_info.get("object_name") == "*"
+                    ):
+                        return False  # Should not find the privilege
+                return True
+            except Exception as e:
+                logger.debug(f"Error checking role privileges v2: {e}")
+                return False
+
+        assert self.wait_for_sync(
+            check_revoke, sync_timeout, f"revoke privilege v2 from role {role_name}"
+        )
+
+    def test_add_privileges_to_group(
+        self, upstream_client, downstream_client, sync_timeout
+    ):
+        """Test ADD_PRIVILEGES_TO_GROUP operation sync."""
+        # Store upstream client for teardown
+        self._upstream_client = upstream_client
+
+        group_name = self.gen_unique_name("test_priv_group_add")
+
+        # Initial cleanup
+        try:
+            upstream_client.drop_privilege_group(group_name)
+        except:
+            pass
+
+        # Create privilege group first
+        upstream_client.create_privilege_group(group_name)
+
+        # Wait for creation to sync
+        def check_create():
+            try:
+                downstream_groups = downstream_client.list_privilege_groups()
+                print(f"DEBUG: downstream_groups in check_create: {downstream_groups}")
+                return group_name in [
+                    group["privilege_group"] for group in downstream_groups
+                ]
+            except Exception as e:
+                logger.debug(f"Error checking privilege groups: {e}")
+                return False
+
+        assert self.wait_for_sync(
+            check_create, sync_timeout, f"create privilege group {group_name}"
+        )
+
+        # Add privileges to group in upstream
+        privileges_to_add = ["Search", "Query"]
+        upstream_client.add_privileges_to_group(group_name, privileges_to_add)
+
+        # Wait for add privileges to sync
+        def check_upstream():
+            client = upstream_client
+            try:
+                groups = client.list_privilege_groups()
+                print(f"DEBUG: groups in check_add: {groups}")
+                assert group_name in [group["privilege_group"] for group in groups]
+                group_info = None
+                for group in groups:
+                    if group["privilege_group"] == group_name:
+                        group_info = group
+                        break
+                assert group_info is not None
+                assert "Search" in group_info["privileges"]
+                assert "Query" in group_info["privileges"]
+                return True
+
+            except Exception as e:
+                logger.debug(f"Error checking privilege groups: {e}")
+                return False
+
+        assert self.wait_for_sync(
+            check_upstream, sync_timeout, f"add privileges to group {group_name}"
+        )
+
+        # Wait for add privileges to sync
+        def check_downstream():
+            client = upstream_client
+            try:
+                groups = client.list_privilege_groups()
+                print(f"DEBUG: groups in check_add: {groups}")
+                assert group_name in [group["privilege_group"] for group in groups]
+                group_info = None
+                for group in groups:
+                    if group["privilege_group"] == group_name:
+                        group_info = group
+                        break
+                assert group_info is not None
+                assert "Search" in group_info["privileges"]
+                assert "Query" in group_info["privileges"]
+                return True
+
+            except Exception as e:
+                logger.debug(f"Error checking privilege groups: {e}")
+                return False
+
+        assert self.wait_for_sync(
+            check_downstream, sync_timeout, f"add privileges to group {group_name}"
+        )
+
+        # Cleanup
+        try:
+            upstream_client.drop_privilege_group(group_name)
+        except:
+            pass
+
+    def test_remove_privileges_from_group(
+        self, upstream_client, downstream_client, sync_timeout
+    ):
+        """Test REMOVE_PRIVILEGES_FROM_GROUP operation sync."""
+        # Store upstream client for teardown
+        self._upstream_client = upstream_client
+
+        group_name = self.gen_unique_name("test_priv_group_add")
+
+        # Initial cleanup
+        try:
+            upstream_client.drop_privilege_group(group_name)
+        except:
+            pass
+
+        # Create privilege group first
+        upstream_client.create_privilege_group(group_name)
+
+        # Wait for creation to sync
+        def check_create():
+            try:
+                downstream_groups = downstream_client.list_privilege_groups()
+                print(f"DEBUG: downstream_groups in check_create: {downstream_groups}")
+                return group_name in [
+                    group["privilege_group"] for group in downstream_groups
+                ]
+            except Exception as e:
+                logger.debug(f"Error checking privilege groups: {e}")
+                return False
+
+        assert self.wait_for_sync(
+            check_create, sync_timeout, f"create privilege group {group_name}"
+        )
+
+        # Add privileges to group in upstream
+        privileges_to_add = ["Search", "Query"]
+        upstream_client.add_privileges_to_group(group_name, privileges_to_add)
+
+        # Wait for add privileges to sync
+        def check_upstream():
+            client = upstream_client
+            try:
+                groups = client.list_privilege_groups()
+                print(f"DEBUG: groups in check_add: {groups}")
+                assert group_name in [group["privilege_group"] for group in groups]
+                group_info = None
+                for group in groups:
+                    if group["privilege_group"] == group_name:
+                        group_info = group
+                        break
+                assert group_info is not None
+                assert "Search" in group_info["privileges"]
+                assert "Query" in group_info["privileges"]
+                return True
+
+            except Exception as e:
+                logger.debug(f"Error checking privilege groups: {e}")
+                return False
+
+        assert self.wait_for_sync(
+            check_upstream, sync_timeout, f"add privileges to group {group_name}"
+        )
+
+        # Wait for add privileges to sync
+        def check_downstream():
+            client = upstream_client
+            try:
+                groups = client.list_privilege_groups()
+                print(f"DEBUG: groups in check_add: {groups}")
+                assert group_name in [group["privilege_group"] for group in groups]
+                group_info = None
+                for group in groups:
+                    if group["privilege_group"] == group_name:
+                        group_info = group
+                        break
+                assert group_info is not None
+                assert "Search" in group_info["privileges"]
+                assert "Query" in group_info["privileges"]
+                return True
+
+            except Exception as e:
+                logger.debug(f"Error checking privilege groups: {e}")
+                return False
+
+        assert self.wait_for_sync(
+            check_downstream, sync_timeout, f"add privileges to group {group_name}"
+        )
+
+        # remove privileges from group in upstream
+        privileges_to_remove = ["Search", "Query"]
+        upstream_client.remove_privileges_from_group(group_name, privileges_to_remove)
+
+        # Wait for remove privileges to sync
+        def check_upstream():
+            client = upstream_client
+            try:
+                groups = client.list_privilege_groups()
+                print(f"DEBUG: groups in check_remove: {groups}")
+                assert group_name in [group["privilege_group"] for group in groups]
+                group_info = None
+                for group in groups:
+                    if group["privilege_group"] == group_name:
+                        group_info = group
+                        break
+                assert group_info is not None
+                assert "Search" not in group_info["privileges"]
+                assert "Query" not in group_info["privileges"]
+                return True
+
+            except Exception as e:
+                logger.debug(f"Error checking privilege groups: {e}")
+                return False
+
+        assert self.wait_for_sync(
+            check_upstream, sync_timeout, f"remove privileges from group {group_name}"
+        )
+
+        # Wait for remove privileges to sync
+        def check_downstream():
+            client = downstream_client
+            try:
+                groups = client.list_privilege_groups()
+                print(f"DEBUG: groups in check_remove: {groups}")
+                assert group_name in [group["privilege_group"] for group in groups]
+                group_info = None
+                for group in groups:
+                    if group["privilege_group"] == group_name:
+                        group_info = group
+                        break
+                assert group_info is not None
+                assert "Search" not in group_info["privileges"]
+                assert "Query" not in group_info["privileges"]
+                return True
+
+            except Exception as e:
+                logger.debug(f"Error checking privilege groups: {e}")
+                return False
+
+        assert self.wait_for_sync(
+            check_downstream, sync_timeout, f"remove privileges from group {group_name}"
+        )

tests/python_client/cdc/testcases/test_resource_group.py

@@ -0,0 +1,157 @@
+"""
+CDC non-replication tests for resource group operations.
+
+Resource groups and replica assignment are per-cluster state; by design
+CDC does NOT propagate RG create/drop/update/transfer-replica to the
+downstream cluster. These tests guard that invariant.
+"""
+
+import time
+
+from .base import TestCDCSyncBase, logger
+
+
+class TestCDCSyncResourceGroup(TestCDCSyncBase):
+    """Verify that resource group operations are NOT replicated by CDC."""
+
+    def setup_method(self):
+        """Setup for each test method."""
+        self.resources_to_cleanup = []
+
+    def teardown_method(self):
+        """Cleanup both upstream and downstream (downstream won't auto-sync)."""
+        upstream_client = getattr(self, "_upstream_client", None)
+        downstream_client = getattr(self, "_downstream_client", None)
+
+        for client in (upstream_client, downstream_client):
+            if not client:
+                continue
+            for resource_type, resource_name in self.resources_to_cleanup:
+                if resource_type == "resource_group":
+                    self._cleanup_resource_group(client, resource_name)
+                elif resource_type == "collection":
+                    self.cleanup_collection(client, resource_name)
+
+    def _cleanup_resource_group(self, client, rg_name):
+        """Clean up resource group if exists (skipping default RG)."""
+        try:
+            existing = client.list_resource_groups()
+            if rg_name in existing:
+                logger.info(f"[CLEANUP] Cleaning up resource group: {rg_name}")
+                client.drop_resource_group(rg_name)
+                logger.info(f"[SUCCESS] Resource group {rg_name} cleaned up successfully")
+            else:
+                logger.debug(f"Resource group {rg_name} does not exist, skipping cleanup")
+        except Exception as e:
+            logger.warning(f"[FAILED] Failed to cleanup resource group {rg_name}: {e}")
+
+    def test_create_resource_group_not_replicated(self, upstream_client, downstream_client, sync_timeout):
+        """Creating an RG on upstream must NOT create it on downstream."""
+        self._upstream_client = upstream_client
+        self._downstream_client = downstream_client
+
+        rg_name = self.gen_unique_name("test_rg_create")
+        self.resources_to_cleanup.append(("resource_group", rg_name))
+
+        self._cleanup_resource_group(upstream_client, rg_name)
+        self._cleanup_resource_group(downstream_client, rg_name)
+
+        upstream_client.create_resource_group(rg_name)
+        assert rg_name in upstream_client.list_resource_groups(), f"Resource group {rg_name} not created in upstream"
+
+        # Wait the full sync window; if CDC were going to leak the RG it would
+        # have shown up by now.
+        time.sleep(sync_timeout)
+        downstream_rgs = downstream_client.list_resource_groups()
+        assert rg_name not in downstream_rgs, (
+            f"Resource group {rg_name} unexpectedly appeared on downstream (RG ops must not replicate)"
+        )
+
+    def test_drop_resource_group_not_replicated(self, upstream_client, downstream_client, sync_timeout):
+        """Dropping an RG on upstream must NOT drop it on downstream."""
+        self._upstream_client = upstream_client
+        self._downstream_client = downstream_client
+
+        rg_name = self.gen_unique_name("test_rg_drop")
+        self.resources_to_cleanup.append(("resource_group", rg_name))
+
+        self._cleanup_resource_group(upstream_client, rg_name)
+        self._cleanup_resource_group(downstream_client, rg_name)
+
+        # Create the RG on BOTH sides (independently, since create isn't replicated either).
+        upstream_client.create_resource_group(rg_name)
+        downstream_client.create_resource_group(rg_name)
+        assert rg_name in upstream_client.list_resource_groups()
+        assert rg_name in downstream_client.list_resource_groups()
+
+        # Drop only on upstream; downstream must retain its own copy.
+        upstream_client.drop_resource_group(rg_name)
+        assert rg_name not in upstream_client.list_resource_groups(), (
+            f"Resource group {rg_name} still exists in upstream after drop"
+        )
+
+        time.sleep(sync_timeout)
+        downstream_rgs = downstream_client.list_resource_groups()
+        assert rg_name in downstream_rgs, (
+            f"Resource group {rg_name} was dropped on downstream after upstream drop "
+            f"(RG ops must not replicate). downstream RGs: {downstream_rgs}"
+        )
+
+    def test_update_resource_group_not_replicated(self, upstream_client, downstream_client, sync_timeout):
+        """Updating RG config on upstream must NOT reconfigure downstream's RG."""
+        self._upstream_client = upstream_client
+        self._downstream_client = downstream_client
+
+        rg_name = self.gen_unique_name("test_rg_update")
+        self.resources_to_cleanup.append(("resource_group", rg_name))
+
+        self._cleanup_resource_group(upstream_client, rg_name)
+        self._cleanup_resource_group(downstream_client, rg_name)
+
+        upstream_config = {"requests": {"node_num": 0}, "limits": {"node_num": 2}}
+        downstream_config = {"requests": {"node_num": 0}, "limits": {"node_num": 1}}
+
+        # Create the RG on both sides with different configs.
+        upstream_client.create_resource_group(rg_name, config=upstream_config)
+        downstream_client.create_resource_group(rg_name, config=downstream_config)
+        downstream_desc_before = downstream_client.describe_resource_group(rg_name)
+        logger.info(f"Downstream RG before upstream update: {downstream_desc_before}")
+
+        # Upstream has already been created with its config; nothing else to
+        # update since config drift itself would be the leak. Wait and confirm
+        # downstream config is unchanged. ResourceGroupInfo has no __eq__, so
+        # compare the str() form (includes config/limits/requests/nodes).
+        time.sleep(sync_timeout)
+        downstream_desc_after = downstream_client.describe_resource_group(rg_name)
+        logger.info(f"Downstream RG after upstream update: {downstream_desc_after}")
+        assert str(downstream_desc_after) == str(downstream_desc_before), (
+            f"Downstream RG {rg_name} was modified by upstream config (RG ops must not replicate). "
+            f"before={downstream_desc_before}, after={downstream_desc_after}"
+        )
+
+    def test_transfer_replica_not_replicated(self, upstream_client, downstream_client, sync_timeout):
+        """Creating multiple RGs on upstream must NOT create any of them on downstream."""
+        self._upstream_client = upstream_client
+        self._downstream_client = downstream_client
+
+        rg_name_1 = self.gen_unique_name("test_rg_transfer_1")
+        rg_name_2 = self.gen_unique_name("test_rg_transfer_2")
+        self.resources_to_cleanup.append(("resource_group", rg_name_1))
+        self.resources_to_cleanup.append(("resource_group", rg_name_2))
+
+        self._cleanup_resource_group(upstream_client, rg_name_1)
+        self._cleanup_resource_group(upstream_client, rg_name_2)
+        self._cleanup_resource_group(downstream_client, rg_name_1)
+        self._cleanup_resource_group(downstream_client, rg_name_2)
+
+        upstream_client.create_resource_group(rg_name_1)
+        upstream_client.create_resource_group(rg_name_2)
+        upstream_rgs = upstream_client.list_resource_groups()
+        assert rg_name_1 in upstream_rgs
+        assert rg_name_2 in upstream_rgs
+
+        time.sleep(sync_timeout)
+        downstream_rgs = downstream_client.list_resource_groups()
+        assert rg_name_1 not in downstream_rgs and rg_name_2 not in downstream_rgs, (
+            f"Upstream RGs leaked to downstream (RG ops must not replicate). downstream RGs: {downstream_rgs}"
+        )

tests/python_client/cdc/testcases/test_schema_features.py

@@ -0,0 +1,690 @@
+"""
+CDC sync tests for advanced schema features (dynamic fields, nullable, default values,
+partition keys, clustering keys, and combinations thereof).
+"""
+
+import random
+import time
+
+from pymilvus import DataType
+
+from .base import TestCDCSyncBase, logger
+
+
+class TestCDCSyncSchemaFeatures(TestCDCSyncBase):
+    """Test CDC sync for advanced schema features."""
+
+    def setup_method(self):
+        """Setup for each test method."""
+        self.resources_to_cleanup = []
+
+    def teardown_method(self):
+        """Cleanup after each test method - only cleanup upstream, downstream will sync."""
+        upstream_client = getattr(self, "_upstream_client", None)
+
+        if upstream_client:
+            for resource_type, resource_name in self.resources_to_cleanup:
+                if resource_type == "collection":
+                    self.cleanup_collection(upstream_client, resource_name)
+
+            time.sleep(1)  # Allow cleanup to sync to downstream
+
+    def test_dynamic_schema_sync(self, upstream_client, downstream_client, sync_timeout):
+        """Test that dynamic schema fields are correctly replicated via CDC."""
+        start_time = time.time()
+        collection_name = self.gen_unique_name("test_dynamic_schema", max_length=50)
+
+        self.log_test_start("test_dynamic_schema_sync", "DYNAMIC_SCHEMA", collection_name)
+        self._upstream_client = upstream_client
+        self.resources_to_cleanup.append(("collection", collection_name))
+
+        try:
+            self.cleanup_collection(upstream_client, collection_name)
+
+            # Create dynamic schema collection
+            self.log_operation("CREATE_COLLECTION", "collection", collection_name, "upstream")
+            upstream_client.create_collection(
+                collection_name=collection_name,
+                schema=self.create_dynamic_schema(upstream_client),
+            )
+
+            # Create HNSW index and load
+            index_params = upstream_client.prepare_index_params()
+            index_params.add_index(
+                field_name="float_vector",
+                index_type="HNSW",
+                metric_type="L2",
+                params={"M": 8, "efConstruction": 64},
+            )
+            upstream_client.create_index(collection_name, index_params)
+            upstream_client.load_collection(collection_name)
+
+            # Wait for collection creation to sync
+            def check_create():
+                return downstream_client.has_collection(collection_name)
+
+            assert self.wait_for_sync(check_create, sync_timeout, f"create collection {collection_name}")
+
+            # Insert 200 rows with extra dynamic fields
+            extra_fields = {"extra_int": int, "extra_str": str, "extra_float": float}
+            test_data = self.generate_dynamic_data(200, extra_fields=extra_fields)
+            self.log_data_operation("INSERT", collection_name, len(test_data), "- dynamic schema data")
+            upstream_client.insert(collection_name, test_data)
+            upstream_client.flush(collection_name)
+
+            # Query extra_int > 0 on upstream
+            self.log_sync_verification("DYNAMIC_SCHEMA", collection_name, "extra_int > 0 count matches downstream")
+            upstream_result = upstream_client.query(
+                collection_name=collection_name,
+                filter="extra_int > 0",
+                output_fields=["count(*)"],
+            )
+            upstream_count = upstream_result[0]["count(*)"] if upstream_result else 0
+            logger.info(f"[UPSTREAM] extra_int > 0 count: {upstream_count}")
+
+            # Wait for sync and verify downstream count matches
+            def check_data():
+                try:
+                    down_result = downstream_client.query(
+                        collection_name=collection_name,
+                        filter="extra_int > 0",
+                        output_fields=["count(*)"],
+                    )
+                    down_count = down_result[0]["count(*)"] if down_result else 0
+                    logger.info(f"[SYNC_PROGRESS] downstream extra_int > 0 count: {down_count}/{upstream_count}")
+                    return down_count == upstream_count
+                except Exception as e:
+                    logger.warning(f"Dynamic schema sync check failed: {e}")
+                    return False
+
+            sync_success = self.wait_for_sync(check_data, sync_timeout, f"dynamic schema data sync {collection_name}")
+            assert sync_success, f"Dynamic schema data failed to sync to downstream for {collection_name}"
+
+            # Verify data sampling for extra fields
+            match, mismatch, details = self.verify_data_sampling(
+                upstream_client,
+                downstream_client,
+                collection_name,
+                sample_ratio=0.1,
+                output_fields=["id", "varchar_field", "extra_int", "extra_str", "extra_float"],
+            )
+            logger.info(f"[VERIFY] Dynamic schema sampling: match={match}, mismatch={mismatch}")
+            assert mismatch == 0, f"Dynamic field data mismatch detected: {details}"
+
+            duration = time.time() - start_time
+            self.log_test_end("test_dynamic_schema_sync", True, duration)
+
+        except Exception as e:
+            duration = time.time() - start_time
+            logger.error(f"[ERROR] test_dynamic_schema_sync failed: {e}")
+            self.log_test_end("test_dynamic_schema_sync", False, duration)
+            raise
+
+    def test_nullable_fields_sync(self, upstream_client, downstream_client, sync_timeout):
+        """Test that nullable field values (including NULLs) are correctly replicated via CDC."""
+        start_time = time.time()
+        collection_name = self.gen_unique_name("test_nullable_flds", max_length=50)
+
+        self.log_test_start("test_nullable_fields_sync", "NULLABLE_FIELDS", collection_name)
+        self._upstream_client = upstream_client
+        self.resources_to_cleanup.append(("collection", collection_name))
+
+        try:
+            self.cleanup_collection(upstream_client, collection_name)
+
+            # Create nullable schema collection
+            self.log_operation("CREATE_COLLECTION", "collection", collection_name, "upstream")
+            upstream_client.create_collection(
+                collection_name=collection_name,
+                schema=self.create_nullable_schema(upstream_client),
+            )
+
+            # Create index and load
+            index_params = upstream_client.prepare_index_params()
+            index_params.add_index(
+                field_name="float_vector",
+                index_type="HNSW",
+                metric_type="L2",
+                params={"M": 8, "efConstruction": 64},
+            )
+            upstream_client.create_index(collection_name, index_params)
+            upstream_client.load_collection(collection_name)
+
+            # Wait for collection creation to sync
+            def check_create():
+                return downstream_client.has_collection(collection_name)
+
+            assert self.wait_for_sync(check_create, sync_timeout, f"create collection {collection_name}")
+
+            # Insert 200 rows with null_ratio=0.3
+            test_data = self.generate_nullable_data(200, null_ratio=0.3)
+            self.log_data_operation("INSERT", collection_name, len(test_data), "- nullable data null_ratio=0.3")
+            upstream_client.insert(collection_name, test_data)
+            upstream_client.flush(collection_name)
+
+            # Count nulls and not-nulls for nullable_int64 on upstream
+            null_result = upstream_client.query(
+                collection_name=collection_name,
+                filter="nullable_int64 is null",
+                output_fields=["count(*)"],
+            )
+            not_null_result = upstream_client.query(
+                collection_name=collection_name,
+                filter="nullable_int64 is not null",
+                output_fields=["count(*)"],
+            )
+            upstream_null_count = null_result[0]["count(*)"] if null_result else 0
+            upstream_not_null_count = not_null_result[0]["count(*)"] if not_null_result else 0
+            logger.info(f"[UPSTREAM] nullable_int64 null={upstream_null_count}, not_null={upstream_not_null_count}")
+
+            self.log_sync_verification("NULLABLE_FIELDS", collection_name, "null counts match downstream")
+
+            # Wait for sync and verify null/not-null counts match on downstream
+            def check_null_counts():
+                try:
+                    d_null = downstream_client.query(
+                        collection_name=collection_name,
+                        filter="nullable_int64 is null",
+                        output_fields=["count(*)"],
+                    )
+                    d_not_null = downstream_client.query(
+                        collection_name=collection_name,
+                        filter="nullable_int64 is not null",
+                        output_fields=["count(*)"],
+                    )
+                    d_null_count = d_null[0]["count(*)"] if d_null else 0
+                    d_not_null_count = d_not_null[0]["count(*)"] if d_not_null else 0
+                    logger.info(
+                        f"[SYNC_PROGRESS] downstream nullable_int64 null={d_null_count}/{upstream_null_count}, "
+                        f"not_null={d_not_null_count}/{upstream_not_null_count}"
+                    )
+                    return d_null_count == upstream_null_count and d_not_null_count == upstream_not_null_count
+                except Exception as e:
+                    logger.warning(f"Nullable sync check failed: {e}")
+                    return False
+
+            sync_success = self.wait_for_sync(
+                check_null_counts, sync_timeout, f"nullable fields sync {collection_name}"
+            )
+            assert sync_success, f"Nullable field counts failed to sync to downstream for {collection_name}"
+
+            duration = time.time() - start_time
+            self.log_test_end("test_nullable_fields_sync", True, duration)
+
+        except Exception as e:
+            duration = time.time() - start_time
+            logger.error(f"[ERROR] test_nullable_fields_sync failed: {e}")
+            self.log_test_end("test_nullable_fields_sync", False, duration)
+            raise
+
+    def test_default_values_sync(self, upstream_client, downstream_client, sync_timeout):
+        """Test that default field values are applied and replicated correctly via CDC."""
+        start_time = time.time()
+        collection_name = self.gen_unique_name("test_default_vals", max_length=50)
+
+        self.log_test_start("test_default_values_sync", "DEFAULT_VALUES", collection_name)
+        self._upstream_client = upstream_client
+        self.resources_to_cleanup.append(("collection", collection_name))
+
+        try:
+            self.cleanup_collection(upstream_client, collection_name)
+
+            # Create default values schema collection
+            self.log_operation("CREATE_COLLECTION", "collection", collection_name, "upstream")
+            upstream_client.create_collection(
+                collection_name=collection_name,
+                schema=self.create_default_values_schema(upstream_client),
+            )
+
+            # Create index and load
+            index_params = upstream_client.prepare_index_params()
+            index_params.add_index(
+                field_name="float_vector",
+                index_type="HNSW",
+                metric_type="L2",
+                params={"M": 8, "efConstruction": 64},
+            )
+            upstream_client.create_index(collection_name, index_params)
+            upstream_client.load_collection(collection_name)
+
+            # Wait for collection creation to sync
+            def check_create():
+                return downstream_client.has_collection(collection_name)
+
+            assert self.wait_for_sync(check_create, sync_timeout, f"create collection {collection_name}")
+
+            # Insert 100 rows providing ONLY float_vector — default fields are omitted
+            test_data = [{"float_vector": [random.random() for _ in range(128)]} for _ in range(100)]
+            self.log_data_operation("INSERT", collection_name, len(test_data), "- only float_vector provided")
+            upstream_client.insert(collection_name, test_data)
+            upstream_client.flush(collection_name)
+
+            # Query default_varchar == "default" on upstream — should be 100
+            upstream_result = upstream_client.query(
+                collection_name=collection_name,
+                filter='default_varchar == "default"',
+                output_fields=["count(*)"],
+            )
+            upstream_count = upstream_result[0]["count(*)"] if upstream_result else 0
+            logger.info(f'[UPSTREAM] default_varchar == "default" count: {upstream_count}')
+            assert upstream_count == 100, f"Expected 100 rows with default varchar on upstream, got {upstream_count}"
+
+            self.log_sync_verification(
+                "DEFAULT_VALUES", collection_name, 'default_varchar == "default" count=100 on downstream'
+            )
+
+            # Wait for sync and verify same count on downstream
+            def check_defaults():
+                try:
+                    down_result = downstream_client.query(
+                        collection_name=collection_name,
+                        filter='default_varchar == "default"',
+                        output_fields=["count(*)"],
+                    )
+                    down_count = down_result[0]["count(*)"] if down_result else 0
+                    logger.info(f"[SYNC_PROGRESS] downstream default_varchar count: {down_count}/100")
+                    return down_count == 100
+                except Exception as e:
+                    logger.warning(f"Default values sync check failed: {e}")
+                    return False
+
+            sync_success = self.wait_for_sync(check_defaults, sync_timeout, f"default values sync {collection_name}")
+            assert sync_success, f"Default value data failed to sync to downstream for {collection_name}"
+
+            duration = time.time() - start_time
+            self.log_test_end("test_default_values_sync", True, duration)
+
+        except Exception as e:
+            duration = time.time() - start_time
+            logger.error(f"[ERROR] test_default_values_sync failed: {e}")
+            self.log_test_end("test_default_values_sync", False, duration)
+            raise
+
+    def test_partition_key_sync(self, upstream_client, downstream_client, sync_timeout):
+        """Test that partition key schema and data are correctly replicated via CDC."""
+        start_time = time.time()
+        collection_name = self.gen_unique_name("test_part_key", max_length=50)
+
+        self.log_test_start("test_partition_key_sync", "PARTITION_KEY", collection_name)
+        self._upstream_client = upstream_client
+        self.resources_to_cleanup.append(("collection", collection_name))
+
+        try:
+            self.cleanup_collection(upstream_client, collection_name)
+
+            # Create partition key schema (VarChar key)
+            self.log_operation("CREATE_COLLECTION", "collection", collection_name, "upstream")
+            upstream_client.create_collection(
+                collection_name=collection_name,
+                schema=self.create_partition_key_schema(upstream_client, key_type="VarChar"),
+            )
+
+            # Create index and load
+            index_params = upstream_client.prepare_index_params()
+            index_params.add_index(
+                field_name="float_vector",
+                index_type="HNSW",
+                metric_type="L2",
+                params={"M": 8, "efConstruction": 64},
+            )
+            upstream_client.create_index(collection_name, index_params)
+            upstream_client.load_collection(collection_name)
+
+            # Wait for collection creation to sync
+            def check_create():
+                return downstream_client.has_collection(collection_name)
+
+            assert self.wait_for_sync(check_create, sync_timeout, f"create collection {collection_name}")
+
+            # Insert 500 rows with random categories as partition key values
+            categories = ["cat_A", "cat_B", "cat_C", "cat_D", "cat_E"]
+            test_data = [
+                {
+                    "float_vector": [random.random() for _ in range(128)],
+                    "partition_key_field": random.choice(categories),
+                    "data_field": f"data_{i}_{random.randint(1000, 9999)}",
+                }
+                for i in range(500)
+            ]
+            self.log_data_operation("INSERT", collection_name, len(test_data), "- partition key data")
+            upstream_client.insert(collection_name, test_data)
+            upstream_client.flush(collection_name)
+
+            self.log_sync_verification(
+                "PARTITION_KEY", collection_name, "count=500 and partition_key field on downstream"
+            )
+
+            # Wait for sync and verify count=500 on downstream
+            def check_data():
+                try:
+                    result = downstream_client.query(
+                        collection_name=collection_name,
+                        filter="",
+                        output_fields=["count(*)"],
+                    )
+                    count = result[0]["count(*)"] if result else 0
+                    logger.info(f"[SYNC_PROGRESS] downstream count: {count}/500")
+                    return count >= 500
+                except Exception as e:
+                    logger.warning(f"Partition key sync check failed: {e}")
+                    return False
+
+            sync_success = self.wait_for_sync(check_data, sync_timeout, f"partition key data sync {collection_name}")
+            assert sync_success, f"Partition key data failed to sync to downstream for {collection_name}"
+
+            # Verify partition_key_field is present in downstream describe_collection
+            downstream_info = downstream_client.describe_collection(collection_name)
+            downstream_fields = [f["name"] for f in downstream_info.get("fields", [])]
+            logger.info(f"[VERIFY] Downstream fields: {downstream_fields}")
+            assert "partition_key_field" in downstream_fields, (
+                f"partition_key_field not found in downstream collection schema: {downstream_fields}"
+            )
+
+            duration = time.time() - start_time
+            self.log_test_end("test_partition_key_sync", True, duration)
+
+        except Exception as e:
+            duration = time.time() - start_time
+            logger.error(f"[ERROR] test_partition_key_sync failed: {e}")
+            self.log_test_end("test_partition_key_sync", False, duration)
+            raise
+
+    def test_clustering_key_sync(self, upstream_client, downstream_client, sync_timeout):
+        """Test that clustering key schema and data are correctly replicated via CDC."""
+        start_time = time.time()
+        collection_name = self.gen_unique_name("test_cluster_key", max_length=50)
+
+        self.log_test_start("test_clustering_key_sync", "CLUSTERING_KEY", collection_name)
+        self._upstream_client = upstream_client
+        self.resources_to_cleanup.append(("collection", collection_name))
+
+        try:
+            self.cleanup_collection(upstream_client, collection_name)
+
+            # Create clustering key schema
+            self.log_operation("CREATE_COLLECTION", "collection", collection_name, "upstream")
+            upstream_client.create_collection(
+                collection_name=collection_name,
+                schema=self.create_clustering_key_schema(upstream_client),
+            )
+
+            # Create index and load
+            index_params = upstream_client.prepare_index_params()
+            index_params.add_index(
+                field_name="float_vector",
+                index_type="HNSW",
+                metric_type="L2",
+                params={"M": 8, "efConstruction": 64},
+            )
+            upstream_client.create_index(collection_name, index_params)
+            upstream_client.load_collection(collection_name)
+
+            # Wait for collection creation to sync
+            def check_create():
+                return downstream_client.has_collection(collection_name)
+
+            assert self.wait_for_sync(check_create, sync_timeout, f"create collection {collection_name}")
+
+            # Insert 300 rows
+            test_data = [
+                {
+                    "float_vector": [random.random() for _ in range(128)],
+                    "clustering_key_field": random.randint(0, 1000),
+                    "data_field": f"data_{i}_{random.randint(1000, 9999)}",
+                }
+                for i in range(300)
+            ]
+            self.log_data_operation("INSERT", collection_name, len(test_data), "- clustering key data")
+            upstream_client.insert(collection_name, test_data)
+            upstream_client.flush(collection_name)
+
+            self.log_sync_verification(
+                "CLUSTERING_KEY", collection_name, "count=300 and clustering_key field on downstream"
+            )
+
+            # Wait for sync and verify count=300 on downstream
+            def check_data():
+                try:
+                    result = downstream_client.query(
+                        collection_name=collection_name,
+                        filter="",
+                        output_fields=["count(*)"],
+                    )
+                    count = result[0]["count(*)"] if result else 0
+                    logger.info(f"[SYNC_PROGRESS] downstream count: {count}/300")
+                    return count >= 300
+                except Exception as e:
+                    logger.warning(f"Clustering key sync check failed: {e}")
+                    return False
+
+            sync_success = self.wait_for_sync(check_data, sync_timeout, f"clustering key data sync {collection_name}")
+            assert sync_success, f"Clustering key data failed to sync to downstream for {collection_name}"
+
+            # Verify clustering_key_field is present in downstream describe_collection
+            downstream_info = downstream_client.describe_collection(collection_name)
+            downstream_fields = [f["name"] for f in downstream_info.get("fields", [])]
+            logger.info(f"[VERIFY] Downstream fields: {downstream_fields}")
+            assert "clustering_key_field" in downstream_fields, (
+                f"clustering_key_field not found in downstream collection schema: {downstream_fields}"
+            )
+
+            duration = time.time() - start_time
+            self.log_test_end("test_clustering_key_sync", True, duration)
+
+        except Exception as e:
+            duration = time.time() - start_time
+            logger.error(f"[ERROR] test_clustering_key_sync failed: {e}")
+            self.log_test_end("test_clustering_key_sync", False, duration)
+            raise
+
+    def test_nullable_with_defaults(self, upstream_client, downstream_client, sync_timeout):
+        """Test that nullable fields combined with default values sync correctly via CDC.
+
+        Inserts 100 rows in three patterns:
+          i%3==0: explicit values provided for both fields
+          i%3==1: None values (explicit null) provided for both fields
+          i%3==2: fields omitted entirely (uses default / null)
+        """
+        start_time = time.time()
+        collection_name = self.gen_unique_name("test_null_default", max_length=50)
+
+        self.log_test_start("test_nullable_with_defaults", "NULLABLE_WITH_DEFAULTS", collection_name)
+        self._upstream_client = upstream_client
+        self.resources_to_cleanup.append(("collection", collection_name))
+
+        try:
+            self.cleanup_collection(upstream_client, collection_name)
+
+            # Build custom schema: nullable_with_default (INT64, nullable, default=42)
+            #                      nullable_no_default   (VARCHAR, nullable)
+            schema = upstream_client.create_schema()
+            schema.add_field("id", DataType.INT64, is_primary=True, auto_id=True)
+            schema.add_field("float_vector", DataType.FLOAT_VECTOR, dim=128)
+            schema.add_field(
+                "nullable_with_default",
+                DataType.INT64,
+                nullable=True,
+                default_value=42,
+            )
+            schema.add_field(
+                "nullable_no_default",
+                DataType.VARCHAR,
+                max_length=256,
+                nullable=True,
+            )
+
+            self.log_operation("CREATE_COLLECTION", "collection", collection_name, "upstream")
+            upstream_client.create_collection(
+                collection_name=collection_name,
+                schema=schema,
+            )
+
+            # Create index and load
+            index_params = upstream_client.prepare_index_params()
+            index_params.add_index(
+                field_name="float_vector",
+                index_type="HNSW",
+                metric_type="L2",
+                params={"M": 8, "efConstruction": 64},
+            )
+            upstream_client.create_index(collection_name, index_params)
+            upstream_client.load_collection(collection_name)
+
+            # Wait for collection creation to sync
+            def check_create():
+                return downstream_client.has_collection(collection_name)
+
+            assert self.wait_for_sync(check_create, sync_timeout, f"create collection {collection_name}")
+
+            # Insert 100 rows across 3 patterns
+            test_data = []
+            for i in range(100):
+                record = {"float_vector": [random.random() for _ in range(128)]}
+                if i % 3 == 0:
+                    # Explicit values
+                    record["nullable_with_default"] = random.randint(100, 999)
+                    record["nullable_no_default"] = f"explicit_{i}"
+                elif i % 3 == 1:
+                    # Explicit None (null)
+                    record["nullable_with_default"] = None
+                    record["nullable_no_default"] = None
+                # i%3==2: omit both fields — server applies default/null
+                test_data.append(record)
+
+            self.log_data_operation("INSERT", collection_name, len(test_data), "- nullable+default mixed pattern")
+            upstream_client.insert(collection_name, test_data)
+            upstream_client.flush(collection_name)
+
+            self.log_sync_verification("NULLABLE_WITH_DEFAULTS", collection_name, "total count=100 on downstream")
+
+            # Wait for sync and verify total count on downstream
+            def check_data():
+                try:
+                    result = downstream_client.query(
+                        collection_name=collection_name,
+                        filter="",
+                        output_fields=["count(*)"],
+                    )
+                    count = result[0]["count(*)"] if result else 0
+                    logger.info(f"[SYNC_PROGRESS] downstream count: {count}/100")
+                    return count >= 100
+                except Exception as e:
+                    logger.warning(f"Nullable+defaults sync check failed: {e}")
+                    return False
+
+            sync_success = self.wait_for_sync(check_data, sync_timeout, f"nullable+defaults sync {collection_name}")
+            assert sync_success, f"Nullable-with-defaults data failed to sync to downstream for {collection_name}"
+
+            duration = time.time() - start_time
+            self.log_test_end("test_nullable_with_defaults", True, duration)
+
+        except Exception as e:
+            duration = time.time() - start_time
+            logger.error(f"[ERROR] test_nullable_with_defaults failed: {e}")
+            self.log_test_end("test_nullable_with_defaults", False, duration)
+            raise
+
+    def test_dynamic_with_partition_key(self, upstream_client, downstream_client, sync_timeout):
+        """Test that dynamic fields combined with a partition key schema sync correctly via CDC."""
+        start_time = time.time()
+        collection_name = self.gen_unique_name("test_dyn_part_key", max_length=50)
+
+        self.log_test_start("test_dynamic_with_partition_key", "DYNAMIC_WITH_PARTITION_KEY", collection_name)
+        self._upstream_client = upstream_client
+        self.resources_to_cleanup.append(("collection", collection_name))
+
+        try:
+            self.cleanup_collection(upstream_client, collection_name)
+
+            # Build schema: enable_dynamic_field=True + VARCHAR partition key
+            schema = upstream_client.create_schema(enable_dynamic_field=True)
+            schema.add_field("id", DataType.INT64, is_primary=True, auto_id=True)
+            schema.add_field("float_vector", DataType.FLOAT_VECTOR, dim=128)
+            schema.add_field(
+                "pk_field",
+                DataType.VARCHAR,
+                max_length=64,
+                is_partition_key=True,
+            )
+
+            self.log_operation("CREATE_COLLECTION", "collection", collection_name, "upstream")
+            upstream_client.create_collection(
+                collection_name=collection_name,
+                schema=schema,
+            )
+
+            # Create index and load
+            index_params = upstream_client.prepare_index_params()
+            index_params.add_index(
+                field_name="float_vector",
+                index_type="HNSW",
+                metric_type="L2",
+                params={"M": 8, "efConstruction": 64},
+            )
+            upstream_client.create_index(collection_name, index_params)
+            upstream_client.load_collection(collection_name)
+
+            # Wait for collection creation to sync
+            def check_create():
+                return downstream_client.has_collection(collection_name)
+
+            assert self.wait_for_sync(check_create, sync_timeout, f"create collection {collection_name}")
+
+            # Insert 300 rows with dynamic fields + partition key values
+            partitions = ["region_A", "region_B", "region_C", "region_D"]
+            test_data = [
+                {
+                    "float_vector": [random.random() for _ in range(128)],
+                    "pk_field": random.choice(partitions),
+                    # dynamic fields
+                    "dynamic_num": random.randint(1, 10000),
+                    "dynamic_tag": f"tag_{i % 10}",
+                    "dynamic_score": random.uniform(0.0, 100.0),
+                }
+                for i in range(300)
+            ]
+            self.log_data_operation("INSERT", collection_name, len(test_data), "- dynamic+partition_key data")
+            upstream_client.insert(collection_name, test_data)
+            upstream_client.flush(collection_name)
+
+            # Query dynamic_num > 0 on upstream
+            upstream_result = upstream_client.query(
+                collection_name=collection_name,
+                filter="dynamic_num > 0",
+                output_fields=["count(*)"],
+            )
+            upstream_count = upstream_result[0]["count(*)"] if upstream_result else 0
+            logger.info(f"[UPSTREAM] dynamic_num > 0 count: {upstream_count}")
+
+            self.log_sync_verification(
+                "DYNAMIC_WITH_PARTITION_KEY",
+                collection_name,
+                f"dynamic_num > 0 count={upstream_count} on downstream",
+            )
+
+            # Wait for sync and verify the count matches on downstream
+            def check_data():
+                try:
+                    down_result = downstream_client.query(
+                        collection_name=collection_name,
+                        filter="dynamic_num > 0",
+                        output_fields=["count(*)"],
+                    )
+                    down_count = down_result[0]["count(*)"] if down_result else 0
+                    logger.info(f"[SYNC_PROGRESS] downstream dynamic_num > 0 count: {down_count}/{upstream_count}")
+                    return down_count == upstream_count
+                except Exception as e:
+                    logger.warning(f"Dynamic+partition_key sync check failed: {e}")
+                    return False
+
+            sync_success = self.wait_for_sync(check_data, sync_timeout, f"dynamic+partition_key sync {collection_name}")
+            assert sync_success, f"Dynamic+partition_key data failed to sync to downstream for {collection_name}"
+
+            duration = time.time() - start_time
+            self.log_test_end("test_dynamic_with_partition_key", True, duration)
+
+        except Exception as e:
+            duration = time.time() - start_time
+            logger.error(f"[ERROR] test_dynamic_with_partition_key failed: {e}")
+            self.log_test_end("test_dynamic_with_partition_key", False, duration)
+            raise

tests/python_client/cdc/testcases/test_search_verification.py

@@ -0,0 +1,570 @@
+"""
+CDC sync tests for search and query result verification across vector types.
+"""
+
+import random
+import time
+
+import pytest
+from pymilvus import AnnSearchRequest, DataType, RRFRanker
+
+from .base import TestCDCSyncBase, logger
+
+# fmt: off
+VECTOR_PARAMS = [
+    ("FLOAT_VECTOR",        "HNSW",                  "COSINE",  128),
+    ("FLOAT_VECTOR",        "IVF_FLAT",              "L2",      128),
+    ("FLOAT16_VECTOR",      "HNSW",                  "L2",       64),
+    ("BFLOAT16_VECTOR",     "HNSW",                  "L2",       64),
+    ("INT8_VECTOR",         "HNSW",                  "COSINE",   64),
+    ("BINARY_VECTOR",       "BIN_FLAT",              "HAMMING", 128),
+    ("SPARSE_FLOAT_VECTOR", "SPARSE_INVERTED_INDEX", "IP",        0),
+]
+# fmt: on
+
+
+class TestCDCSyncSearchVerification(TestCDCSyncBase):
+    """Test CDC sync for search and query result verification across vector types."""
+
+    def setup_method(self):
+        """Setup for each test method."""
+        self.resources_to_cleanup = []
+
+    def teardown_method(self):
+        """Cleanup after each test method - only cleanup upstream, downstream will sync."""
+        upstream_client = getattr(self, "_upstream_client", None)
+
+        if upstream_client:
+            for resource_type, resource_name in self.resources_to_cleanup:
+                if resource_type == "collection":
+                    self.cleanup_collection(upstream_client, resource_name)
+
+            time.sleep(1)  # Allow cleanup to sync to downstream
+
+    # -------------------------------------------------------------------------
+    # Internal helper
+    # -------------------------------------------------------------------------
+
+    def _setup_collection(self, client, c_name, vector_type, index_type, metric, dim):
+        """
+        Create a single-vector-schema collection, insert 500 records,
+        create an index, and load.
+
+        Returns the collection name (same as c_name).
+        """
+        schema = self.create_single_vector_schema(client, vector_type=vector_type, dim=dim)
+        client.create_collection(collection_name=c_name, schema=schema)
+
+        # Insert 500 records
+        data = self.generate_single_vector_data(500, vector_type=vector_type, dim=dim)
+        client.insert(c_name, data)
+        client.flush(c_name)
+
+        # Build index
+        index_params = client.prepare_index_params()
+        if index_type == "IVF_FLAT":
+            idx_params = {"nlist": 64}
+        elif index_type == "HNSW":
+            idx_params = {"M": 16, "efConstruction": 200}
+        else:
+            idx_params = {}
+
+        index_params.add_index(
+            field_name="vector",
+            index_type=index_type,
+            metric_type=metric,
+            params=idx_params,
+        )
+        client.create_index(c_name, index_params)
+        client.load_collection(c_name)
+
+        return c_name
+
+    # -------------------------------------------------------------------------
+    # Tests
+    # -------------------------------------------------------------------------
+
+    @pytest.mark.parametrize(
+        "vector_type,index_type,metric,dim",
+        VECTOR_PARAMS,
+        ids=[p[0] + "_" + p[1] for p in VECTOR_PARAMS],
+    )
+    def test_search_result_consistency(
+        self,
+        upstream_client,
+        downstream_client,
+        sync_timeout,
+        vector_type,
+        index_type,
+        metric,
+        dim,
+    ):
+        """Verify that ANN search results are consistent between upstream and downstream."""
+        start_time = time.time()
+        c_name = self.gen_unique_name(f"test_src_{vector_type[:4].lower()}", max_length=50)
+
+        self.log_test_start(
+            "test_search_result_consistency",
+            f"SEARCH/{vector_type}/{index_type}",
+            c_name,
+        )
+        self._upstream_client = upstream_client
+        self.resources_to_cleanup.append(("collection", c_name))
+
+        try:
+            self.cleanup_collection(upstream_client, c_name)
+            self._setup_collection(upstream_client, c_name, vector_type, index_type, metric, dim)
+
+            # Wait for at least 500 records to appear on downstream
+            def check_sync():
+                try:
+                    res = downstream_client.query(
+                        collection_name=c_name,
+                        filter="",
+                        output_fields=["count(*)"],
+                    )
+                    cnt = res[0]["count(*)"] if res else 0
+                    logger.info(f"[SYNC_PROGRESS] downstream count: {cnt}/500")
+                    return cnt >= 500
+                except Exception as e:
+                    logger.warning(f"Sync check failed: {e}")
+                    return False
+
+            assert self.wait_for_sync(check_sync, sync_timeout, f"data sync 500 records {c_name}"), (
+                f"Downstream did not receive 500 records within {sync_timeout}s"
+            )
+
+            # Build 5 random query vectors
+            dtype = getattr(DataType, vector_type)
+            query_vectors = self._gen_vectors(5, dim if dim > 0 else 1000, dtype)
+
+            avg_overlap, _, _ = self.verify_search_consistency(
+                upstream_client,
+                downstream_client,
+                c_name,
+                query_vectors,
+                anns_field="vector",
+                limit=10,
+                metric_type=metric,
+            )
+
+            assert avg_overlap >= self.SEARCH_OVERLAP_THRESHOLD, (
+                f"Search overlap {avg_overlap:.4f} is below threshold "
+                f"{self.SEARCH_OVERLAP_THRESHOLD} for {vector_type}/{index_type}"
+            )
+
+        finally:
+            self.log_test_end(
+                "test_search_result_consistency",
+                True,
+                time.time() - start_time,
+            )
+
+    @pytest.mark.parametrize(
+        "vector_type,index_type,metric,dim",
+        VECTOR_PARAMS,
+        ids=[p[0] + "_" + p[1] for p in VECTOR_PARAMS],
+    )
+    def test_query_data_sampling(
+        self,
+        upstream_client,
+        downstream_client,
+        sync_timeout,
+        vector_type,
+        index_type,
+        metric,
+        dim,
+    ):
+        """Verify scalar field values are identical on both sides via random sampling."""
+        start_time = time.time()
+        c_name = self.gen_unique_name(f"test_qds_{vector_type[:4].lower()}", max_length=50)
+
+        self.log_test_start(
+            "test_query_data_sampling",
+            f"QUERY_SAMPLE/{vector_type}/{index_type}",
+            c_name,
+        )
+        self._upstream_client = upstream_client
+        self.resources_to_cleanup.append(("collection", c_name))
+
+        try:
+            self.cleanup_collection(upstream_client, c_name)
+            self._setup_collection(upstream_client, c_name, vector_type, index_type, metric, dim)
+
+            def check_sync():
+                try:
+                    res = downstream_client.query(
+                        collection_name=c_name,
+                        filter="",
+                        output_fields=["count(*)"],
+                    )
+                    cnt = res[0]["count(*)"] if res else 0
+                    logger.info(f"[SYNC_PROGRESS] downstream count: {cnt}/500")
+                    return cnt >= 500
+                except Exception as e:
+                    logger.warning(f"Sync check failed: {e}")
+                    return False
+
+            assert self.wait_for_sync(check_sync, sync_timeout, f"data sync 500 records {c_name}"), (
+                f"Downstream did not receive 500 records within {sync_timeout}s"
+            )
+
+            output_fields = ["id", "int_field", "varchar_field", "float_field"]
+            match_count, mismatch_count, mismatch_details = self.verify_data_sampling(
+                upstream_client,
+                downstream_client,
+                c_name,
+                sample_ratio=0.2,
+                output_fields=output_fields,
+            )
+
+            logger.info(
+                f"[RESULT] Sampling — match={match_count}, mismatch={mismatch_count}, details={mismatch_details[:3]}"
+            )
+            assert mismatch_count == 0, f"Found {mismatch_count} mismatched records: {mismatch_details[:5]}"
+
+        finally:
+            self.log_test_end(
+                "test_query_data_sampling",
+                True,
+                time.time() - start_time,
+            )
+
+    def test_hybrid_search_consistency(
+        self,
+        upstream_client,
+        downstream_client,
+        sync_timeout,
+    ):
+        """Verify hybrid search (dense + sparse, RRF ranker) results are consistent."""
+        start_time = time.time()
+        c_name = self.gen_unique_name("test_hybrid_srch", max_length=50)
+
+        self.log_test_start("test_hybrid_search_consistency", "HYBRID_SEARCH", c_name)
+        self._upstream_client = upstream_client
+        self.resources_to_cleanup.append(("collection", c_name))
+
+        try:
+            self.cleanup_collection(upstream_client, c_name)
+
+            # Build schema: dense FloatVector(128) + sparse
+            schema = upstream_client.create_schema(enable_dynamic_field=True)
+            schema.add_field("id", DataType.INT64, is_primary=True, auto_id=True)
+            schema.add_field("dense", DataType.FLOAT_VECTOR, dim=128)
+            schema.add_field("sparse", DataType.SPARSE_FLOAT_VECTOR)
+            schema.add_field("int_field", DataType.INT64)
+            schema.add_field("varchar_field", DataType.VARCHAR, max_length=256)
+
+            upstream_client.create_collection(collection_name=c_name, schema=schema)
+
+            # Insert 300 records
+            dense_vecs = self._gen_vectors(300, 128, DataType.FLOAT_VECTOR)
+            sparse_vecs = self._gen_vectors(300, 1000, DataType.SPARSE_FLOAT_VECTOR)
+            data = [
+                {
+                    "dense": dense_vecs[i],
+                    "sparse": sparse_vecs[i],
+                    "int_field": random.randint(0, 1000),
+                    "varchar_field": f"hybrid_{i}_{random.randint(1000, 9999)}",
+                }
+                for i in range(300)
+            ]
+            upstream_client.insert(c_name, data)
+            upstream_client.flush(c_name)
+
+            # Create indexes
+            index_params = upstream_client.prepare_index_params()
+            index_params.add_index(
+                field_name="dense",
+                index_type="HNSW",
+                metric_type="COSINE",
+                params={"M": 16, "efConstruction": 200},
+            )
+            index_params.add_index(
+                field_name="sparse",
+                index_type="SPARSE_INVERTED_INDEX",
+                metric_type="IP",
+                params={},
+            )
+            upstream_client.create_index(c_name, index_params)
+            upstream_client.load_collection(c_name)
+
+            # Wait for downstream sync
+            def check_sync():
+                try:
+                    res = downstream_client.query(
+                        collection_name=c_name,
+                        filter="",
+                        output_fields=["count(*)"],
+                    )
+                    cnt = res[0]["count(*)"] if res else 0
+                    logger.info(f"[SYNC_PROGRESS] downstream count: {cnt}/300")
+                    return cnt >= 300
+                except Exception as e:
+                    logger.warning(f"Sync check failed: {e}")
+                    return False
+
+            assert self.wait_for_sync(check_sync, sync_timeout, f"hybrid data sync {c_name}"), (
+                f"Downstream did not receive 300 records within {sync_timeout}s"
+            )
+
+            # Build hybrid search requests
+            q_dense = self._gen_vectors(1, 128, DataType.FLOAT_VECTOR)[0]
+            q_sparse = self._gen_vectors(1, 1000, DataType.SPARSE_FLOAT_VECTOR)[0]
+
+            dense_req = AnnSearchRequest(
+                data=[q_dense],
+                anns_field="dense",
+                param={"metric_type": "COSINE", "params": {"ef": 64}},
+                limit=10,
+            )
+            sparse_req = AnnSearchRequest(
+                data=[q_sparse],
+                anns_field="sparse",
+                param={"metric_type": "IP"},
+                limit=10,
+            )
+
+            up_results = upstream_client.hybrid_search(
+                collection_name=c_name,
+                reqs=[dense_req, sparse_req],
+                ranker=RRFRanker(),
+                limit=10,
+                output_fields=["id"],
+            )
+            down_results = downstream_client.hybrid_search(
+                collection_name=c_name,
+                reqs=[dense_req, sparse_req],
+                ranker=RRFRanker(),
+                limit=10,
+                output_fields=["id"],
+            )
+
+            up_pks = set(hit["id"] for hit in up_results[0]) if up_results else set()
+            down_pks = set(hit["id"] for hit in down_results[0]) if down_results else set()
+            union_size = len(up_pks | down_pks)
+            overlap = len(up_pks & down_pks) / union_size if union_size > 0 else 1.0
+
+            logger.info(f"[RESULT] Hybrid search PK overlap={overlap:.4f} (up={len(up_pks)}, down={len(down_pks)})")
+            assert overlap >= self.SEARCH_OVERLAP_THRESHOLD, (
+                f"Hybrid search overlap {overlap:.4f} below threshold {self.SEARCH_OVERLAP_THRESHOLD}"
+            )
+
+        finally:
+            self.log_test_end(
+                "test_hybrid_search_consistency",
+                True,
+                time.time() - start_time,
+            )
+
+    def test_search_iterator_consistency(
+        self,
+        upstream_client,
+        downstream_client,
+        sync_timeout,
+    ):
+        """Verify search iterator returns the same PK set on upstream and downstream."""
+        start_time = time.time()
+        c_name = self.gen_unique_name("test_srch_iter", max_length=50)
+
+        self.log_test_start("test_search_iterator_consistency", "SEARCH_ITERATOR", c_name)
+        self._upstream_client = upstream_client
+        self.resources_to_cleanup.append(("collection", c_name))
+
+        try:
+            self.cleanup_collection(upstream_client, c_name)
+            self._setup_collection(upstream_client, c_name, "FLOAT_VECTOR", "HNSW", "COSINE", 128)
+
+            def check_sync():
+                try:
+                    res = downstream_client.query(
+                        collection_name=c_name,
+                        filter="",
+                        output_fields=["count(*)"],
+                    )
+                    cnt = res[0]["count(*)"] if res else 0
+                    logger.info(f"[SYNC_PROGRESS] downstream count: {cnt}/500")
+                    return cnt >= 500
+                except Exception as e:
+                    logger.warning(f"Sync check failed: {e}")
+                    return False
+
+            assert self.wait_for_sync(check_sync, sync_timeout, f"data sync 500 records {c_name}"), (
+                f"Downstream did not receive 500 records within {sync_timeout}s"
+            )
+
+            query_vec = self._gen_vectors(1, 128, DataType.FLOAT_VECTOR)[0]
+            search_params = {"metric_type": "COSINE", "params": {"ef": 64}}
+
+            def _collect_iterator_pks(client):
+                pks = set()
+                iterator = client.search_iterator(
+                    collection_name=c_name,
+                    data=[query_vec],
+                    anns_field="vector",
+                    batch_size=50,
+                    limit=200,
+                    param=search_params,
+                    output_fields=["id"],
+                )
+                while True:
+                    batch = iterator.next()
+                    if not batch:
+                        iterator.close()
+                        break
+                    for hit in batch:
+                        pks.add(hit["id"])
+                return pks
+
+            up_pks = _collect_iterator_pks(upstream_client)
+            down_pks = _collect_iterator_pks(downstream_client)
+            union_size = len(up_pks | down_pks)
+            overlap = len(up_pks & down_pks) / union_size if union_size > 0 else 1.0
+
+            logger.info(f"[RESULT] Search iterator overlap={overlap:.4f} (up={len(up_pks)}, down={len(down_pks)})")
+            assert overlap >= self.SEARCH_OVERLAP_THRESHOLD, (
+                f"Search iterator PK overlap {overlap:.4f} below threshold {self.SEARCH_OVERLAP_THRESHOLD}"
+            )
+
+        finally:
+            self.log_test_end(
+                "test_search_iterator_consistency",
+                True,
+                time.time() - start_time,
+            )
+
+    def test_query_iterator_consistency(
+        self,
+        upstream_client,
+        downstream_client,
+        sync_timeout,
+    ):
+        """Verify that a query iterator retrieves identical PK sets from both sides."""
+        start_time = time.time()
+        c_name = self.gen_unique_name("test_qry_iter", max_length=50)
+
+        self.log_test_start("test_query_iterator_consistency", "QUERY_ITERATOR", c_name)
+        self._upstream_client = upstream_client
+        self.resources_to_cleanup.append(("collection", c_name))
+
+        try:
+            self.cleanup_collection(upstream_client, c_name)
+            self._setup_collection(upstream_client, c_name, "FLOAT_VECTOR", "HNSW", "COSINE", 128)
+
+            def check_sync():
+                try:
+                    res = downstream_client.query(
+                        collection_name=c_name,
+                        filter="",
+                        output_fields=["count(*)"],
+                    )
+                    cnt = res[0]["count(*)"] if res else 0
+                    logger.info(f"[SYNC_PROGRESS] downstream count: {cnt}/500")
+                    return cnt >= 500
+                except Exception as e:
+                    logger.warning(f"Sync check failed: {e}")
+                    return False
+
+            assert self.wait_for_sync(check_sync, sync_timeout, f"data sync 500 records {c_name}"), (
+                f"Downstream did not receive 500 records within {sync_timeout}s"
+            )
+
+            up_count, down_count, match = self.verify_iterator_consistency(
+                upstream_client,
+                downstream_client,
+                c_name,
+                batch_size=100,
+            )
+
+            logger.info(f"[RESULT] Query iterator — upstream={up_count}, downstream={down_count}, match={match}")
+            assert match, f"Query iterator PK sets differ: upstream={up_count}, downstream={down_count}"
+
+        finally:
+            self.log_test_end(
+                "test_query_iterator_consistency",
+                True,
+                time.time() - start_time,
+            )
+
+    def test_search_with_filter_consistency(
+        self,
+        upstream_client,
+        downstream_client,
+        sync_timeout,
+    ):
+        """Verify filtered search produces consistent results honoring the filter predicate."""
+        start_time = time.time()
+        c_name = self.gen_unique_name("test_srch_filter", max_length=50)
+
+        self.log_test_start("test_search_with_filter_consistency", "SEARCH_WITH_FILTER", c_name)
+        self._upstream_client = upstream_client
+        self.resources_to_cleanup.append(("collection", c_name))
+
+        try:
+            self.cleanup_collection(upstream_client, c_name)
+            self._setup_collection(upstream_client, c_name, "FLOAT_VECTOR", "HNSW", "COSINE", 128)
+
+            def check_sync():
+                try:
+                    res = downstream_client.query(
+                        collection_name=c_name,
+                        filter="",
+                        output_fields=["count(*)"],
+                    )
+                    cnt = res[0]["count(*)"] if res else 0
+                    logger.info(f"[SYNC_PROGRESS] downstream count: {cnt}/500")
+                    return cnt >= 500
+                except Exception as e:
+                    logger.warning(f"Sync check failed: {e}")
+                    return False
+
+            assert self.wait_for_sync(check_sync, sync_timeout, f"data sync 500 records {c_name}"), (
+                f"Downstream did not receive 500 records within {sync_timeout}s"
+            )
+
+            filter_expr = "int_field > 500"
+            query_vec = self._gen_vectors(1, 128, DataType.FLOAT_VECTOR)[0]
+            search_params = {"metric_type": "COSINE"}
+
+            up_results = upstream_client.search(
+                collection_name=c_name,
+                data=[query_vec],
+                anns_field="vector",
+                search_params=search_params,
+                filter=filter_expr,
+                limit=10,
+                output_fields=["id", "int_field"],
+            )
+            down_results = downstream_client.search(
+                collection_name=c_name,
+                data=[query_vec],
+                anns_field="vector",
+                search_params=search_params,
+                filter=filter_expr,
+                limit=10,
+                output_fields=["id", "int_field"],
+            )
+
+            # Verify filter is honoured on both sides
+            for hit in up_results[0] if up_results else []:
+                assert hit["int_field"] > 500, f"Filter violated on upstream: int_field={hit['int_field']}"
+            for hit in down_results[0] if down_results else []:
+                assert hit["int_field"] > 500, f"Filter violated on downstream: int_field={hit['int_field']}"
+
+            # Verify PK overlap
+            up_pks = set(hit["id"] for hit in up_results[0]) if up_results else set()
+            down_pks = set(hit["id"] for hit in down_results[0]) if down_results else set()
+            union_size = len(up_pks | down_pks)
+            overlap = len(up_pks & down_pks) / union_size if union_size > 0 else 1.0
+
+            logger.info(f"[RESULT] Filtered search overlap={overlap:.4f} (up={len(up_pks)}, down={len(down_pks)})")
+            assert overlap >= self.SEARCH_OVERLAP_THRESHOLD, (
+                f"Filtered search overlap {overlap:.4f} below threshold {self.SEARCH_OVERLAP_THRESHOLD}"
+            )
+
+        finally:
+            self.log_test_end(
+                "test_search_with_filter_consistency",
+                True,
+                time.time() - start_time,
+            )

tests/python_client/cdc/testcases/test_setup_cdc.py

@@ -0,0 +1,412 @@
+"""
+CDC topology setup and configuration test cases.
+"""
+
+import time
+
+import pytest
+from common.common_type import CaseLabel
+
+from cdc.conftest import CDC_UPDATE_REPLICATE_TIMEOUT_SECONDS, apply_replicate_configuration
+
+from .base import TestCDCSyncBase
+
+
+@pytest.mark.tags(CaseLabel.CDC)
+@pytest.mark.skip(reason="Skip topology setup test")
+class TestCDCTopologySetup(TestCDCSyncBase):
+    """Test CDC topology setup, switching, and configuration management."""
+
+    def setup_method(self):
+        """Setup for each test method."""
+        self.resources_to_cleanup = []
+
+    def teardown_method(self):
+        """Cleanup after each test method."""
+        # Cleanup will be handled by individual test methods as needed
+        pass
+
+    def test_normal_topology_setup(
+        self,
+        upstream_client,
+        downstream_client,
+        upstream_uri,
+        downstream_uri,
+        upstream_token,
+        downstream_token,
+        source_cluster_id,
+        target_cluster_id,
+        pchannel_num,
+    ):
+        """Test case 1: Normal upstream/downstream topology setup."""
+        # Create a basic topology configuration
+        config = {
+            "clusters": [
+                {
+                    "cluster_id": source_cluster_id,
+                    "connection_param": {"uri": upstream_uri, "token": upstream_token},
+                    "pchannels": [f"{source_cluster_id}-rootcoord-dml_{i}" for i in range(pchannel_num)],
+                },
+                {
+                    "cluster_id": target_cluster_id,
+                    "connection_param": {
+                        "uri": downstream_uri,
+                        "token": downstream_token,
+                    },
+                    "pchannels": [f"{target_cluster_id}-rootcoord-dml_{i}" for i in range(pchannel_num)],
+                },
+            ],
+            "cross_cluster_topology": [
+                {
+                    "source_cluster_id": source_cluster_id,
+                    "target_cluster_id": target_cluster_id,
+                }
+            ],
+        }
+
+        # Test normal topology setup
+        apply_replicate_configuration([(upstream_client, config), (downstream_client, config)])
+
+        # Wait for configuration to take effect
+        time.sleep(3)
+
+        # Verify topology is working by creating a test collection
+        test_collection_name = self.gen_unique_name("topology_test")
+        self.resources_to_cleanup.append(("collection", test_collection_name))
+
+        # Create collection on upstream
+        upstream_client.create_collection(
+            collection_name=test_collection_name,
+            schema=self.create_default_schema(upstream_client),
+        )
+
+        # Verify it syncs to downstream
+        def check_sync():
+            return downstream_client.has_collection(test_collection_name)
+
+        assert self.wait_for_sync(check_sync, 30, f"collection {test_collection_name} sync")
+
+        # Cleanup
+        self.cleanup_collection(upstream_client, test_collection_name)
+
+    def test_switch_upstream_downstream(
+        self,
+        upstream_client,
+        downstream_client,
+        upstream_uri,
+        downstream_uri,
+        upstream_token,
+        downstream_token,
+        source_cluster_id,
+        target_cluster_id,
+        pchannel_num,
+    ):
+        """Test case 2: Switch upstream/downstream after initial setup."""
+        # First setup normal topology (source -> target)
+        original_config = {
+            "clusters": [
+                {
+                    "cluster_id": source_cluster_id,
+                    "connection_param": {"uri": upstream_uri, "token": upstream_token},
+                    "pchannels": [f"{source_cluster_id}-rootcoord-dml_{i}" for i in range(pchannel_num)],
+                },
+                {
+                    "cluster_id": target_cluster_id,
+                    "connection_param": {
+                        "uri": downstream_uri,
+                        "token": downstream_token,
+                    },
+                    "pchannels": [f"{target_cluster_id}-rootcoord-dml_{i}" for i in range(pchannel_num)],
+                },
+            ],
+            "cross_cluster_topology": [
+                {
+                    "source_cluster_id": source_cluster_id,
+                    "target_cluster_id": target_cluster_id,
+                }
+            ],
+        }
+
+        apply_replicate_configuration([(upstream_client, original_config), (downstream_client, original_config)])
+        time.sleep(3)
+
+        # Now switch the direction (target -> source)
+        switched_config = {
+            "clusters": [
+                {
+                    "cluster_id": source_cluster_id,
+                    "connection_param": {"uri": upstream_uri, "token": upstream_token},
+                    "pchannels": [f"{source_cluster_id}-rootcoord-dml_{i}" for i in range(pchannel_num)],
+                },
+                {
+                    "cluster_id": target_cluster_id,
+                    "connection_param": {
+                        "uri": downstream_uri,
+                        "token": downstream_token,
+                    },
+                    "pchannels": [f"{target_cluster_id}-rootcoord-dml_{i}" for i in range(pchannel_num)],
+                },
+            ],
+            "cross_cluster_topology": [
+                {
+                    "source_cluster_id": target_cluster_id,  # Switched
+                    "target_cluster_id": source_cluster_id,  # Switched
+                }
+            ],
+        }
+
+        # Apply switched configuration
+        apply_replicate_configuration([(upstream_client, switched_config), (downstream_client, switched_config)])
+        time.sleep(3)
+
+        # Test the switched topology by creating collection on downstream (now source)
+        test_collection_name = self.gen_unique_name("switch_test")
+        self.resources_to_cleanup.append(("collection", test_collection_name))
+
+        downstream_client.create_collection(
+            collection_name=test_collection_name,
+            schema=self.create_default_schema(downstream_client),
+        )
+
+        # Verify it syncs to upstream (now target)
+        def check_switched_sync():
+            return upstream_client.has_collection(test_collection_name)
+
+        assert self.wait_for_sync(check_switched_sync, 30, f"switched collection {test_collection_name} sync")
+
+        # Cleanup
+        self.cleanup_collection(downstream_client, test_collection_name)
+
+    def test_invalid_topology_structures(
+        self,
+        upstream_client,
+        downstream_client,
+        upstream_uri,
+        downstream_uri,
+        upstream_token,
+        downstream_token,
+        source_cluster_id,
+        target_cluster_id,
+    ):
+        """Test case 4: Invalid topology structure cases."""
+
+        # Test case 4.1: Missing cluster in cross_cluster_topology
+        invalid_config_1 = {
+            "clusters": [
+                {
+                    "cluster_id": source_cluster_id,
+                    "connection_param": {"uri": upstream_uri, "token": upstream_token},
+                    "pchannels": [f"{source_cluster_id}-rootcoord-dml_0"],
+                }
+            ],
+            "cross_cluster_topology": [
+                {
+                    "source_cluster_id": source_cluster_id,
+                    "target_cluster_id": "nonexistent_cluster",  # This cluster is not defined
+                }
+            ],
+        }
+
+        with pytest.raises(Exception):
+            upstream_client.update_replicate_configuration(
+                timeout=CDC_UPDATE_REPLICATE_TIMEOUT_SECONDS, **invalid_config_1
+            )
+
+        # Test case 4.2: Circular dependency (A -> B, B -> A)
+        invalid_config_2 = {
+            "clusters": [
+                {
+                    "cluster_id": source_cluster_id,
+                    "connection_param": {"uri": upstream_uri, "token": upstream_token},
+                    "pchannels": [f"{source_cluster_id}-rootcoord-dml_0"],
+                },
+                {
+                    "cluster_id": target_cluster_id,
+                    "connection_param": {
+                        "uri": downstream_uri,
+                        "token": downstream_token,
+                    },
+                    "pchannels": [f"{target_cluster_id}-rootcoord-dml_0"],
+                },
+            ],
+            "cross_cluster_topology": [
+                {
+                    "source_cluster_id": source_cluster_id,
+                    "target_cluster_id": target_cluster_id,
+                },
+                {
+                    "source_cluster_id": target_cluster_id,
+                    "target_cluster_id": source_cluster_id,  # Circular dependency
+                },
+            ],
+        }
+
+        # This may or may not fail depending on implementation, but test it
+        with pytest.raises(Exception):
+            upstream_client.update_replicate_configuration(
+                timeout=CDC_UPDATE_REPLICATE_TIMEOUT_SECONDS, **invalid_config_2
+            )
+
+        # Test case 4.3: Invalid connection parameters
+        invalid_config_3 = {
+            "clusters": [
+                {
+                    "cluster_id": source_cluster_id,
+                    "connection_param": {
+                        "uri": "http://invalid-host:19530",  # Invalid URI
+                        "token": "invalid:token",
+                    },
+                    "pchannels": [f"{source_cluster_id}-rootcoord-dml_0"],
+                },
+                {
+                    "cluster_id": target_cluster_id,
+                    "connection_param": {
+                        "uri": downstream_uri,
+                        "token": downstream_token,
+                    },
+                    "pchannels": [f"{target_cluster_id}-rootcoord-dml_0"],
+                },
+            ],
+            "cross_cluster_topology": [
+                {
+                    "source_cluster_id": source_cluster_id,
+                    "target_cluster_id": target_cluster_id,
+                }
+            ],
+        }
+        with pytest.raises(Exception):
+            upstream_client.update_replicate_configuration(
+                timeout=CDC_UPDATE_REPLICATE_TIMEOUT_SECONDS, **invalid_config_3
+            )
+
+        # Test case 4.4: Empty cluster list but non-empty topology
+        invalid_config_4 = {
+            "clusters": [],  # Empty clusters
+            "cross_cluster_topology": [
+                {
+                    "source_cluster_id": source_cluster_id,
+                    "target_cluster_id": target_cluster_id,
+                }
+            ],
+        }
+        with pytest.raises(Exception):
+            upstream_client.update_replicate_configuration(
+                timeout=CDC_UPDATE_REPLICATE_TIMEOUT_SECONDS, **invalid_config_4
+            )
+
+        # Test case 4.5: Invalid pchannel format
+        invalid_config_5 = {
+            "clusters": [
+                {
+                    "cluster_id": source_cluster_id,
+                    "connection_param": {"uri": upstream_uri, "token": upstream_token},
+                    "pchannels": ["invalid-channel-format"],  # Invalid format
+                }
+            ],
+            "cross_cluster_topology": [],
+        }
+        with pytest.raises(Exception):
+            upstream_client.update_replicate_configuration(
+                timeout=CDC_UPDATE_REPLICATE_TIMEOUT_SECONDS, **invalid_config_5
+            )
+
+    @pytest.mark.order(-1)
+    def test_clear_configuration_disconnect(
+        self,
+        upstream_client,
+        downstream_client,
+        upstream_uri,
+        downstream_uri,
+        upstream_token,
+        downstream_token,
+        source_cluster_id,
+        target_cluster_id,
+        pchannel_num,
+    ):
+        """Test case 3: Clear configuration and disconnect CDC."""
+        # First setup normal topology
+        config = {
+            "clusters": [
+                {
+                    "cluster_id": source_cluster_id,
+                    "connection_param": {"uri": upstream_uri, "token": upstream_token},
+                    "pchannels": [f"{source_cluster_id}-rootcoord-dml_{i}" for i in range(pchannel_num)],
+                },
+                {
+                    "cluster_id": target_cluster_id,
+                    "connection_param": {
+                        "uri": downstream_uri,
+                        "token": downstream_token,
+                    },
+                    "pchannels": [f"{target_cluster_id}-rootcoord-dml_{i}" for i in range(pchannel_num)],
+                },
+            ],
+            "cross_cluster_topology": [
+                {
+                    "source_cluster_id": source_cluster_id,
+                    "target_cluster_id": target_cluster_id,
+                }
+            ],
+        }
+
+        apply_replicate_configuration([(upstream_client, config), (downstream_client, config)])
+        time.sleep(3)
+
+        # Verify topology is working
+        test_collection_name = self.gen_unique_name("clear_config_test")
+        upstream_client.create_collection(
+            collection_name=test_collection_name,
+            schema=self.create_default_schema(upstream_client),
+        )
+
+        def check_initial_sync():
+            return downstream_client.has_collection(test_collection_name)
+
+        assert self.wait_for_sync(check_initial_sync, 30, f"initial collection {test_collection_name} sync")
+
+        # Now clear the configuration (empty topology)
+        empty_upstream_config = {
+            "clusters": [
+                {
+                    "cluster_id": source_cluster_id,
+                    "connection_param": {"uri": upstream_uri, "token": upstream_token},
+                    "pchannels": [f"{source_cluster_id}-rootcoord-dml_{i}" for i in range(pchannel_num)],
+                }
+            ],
+            "cross_cluster_topology": [],
+        }
+        empty_downstream_config = {
+            "clusters": [
+                {
+                    "cluster_id": target_cluster_id,
+                    "connection_param": {
+                        "uri": downstream_uri,
+                        "token": downstream_token,
+                    },
+                    "pchannels": [f"{target_cluster_id}-rootcoord-dml_{i}" for i in range(pchannel_num)],
+                }
+            ],
+            "cross_cluster_topology": [],
+        }
+        # Apply empty configuration to disconnect CDC
+        apply_replicate_configuration(
+            [(upstream_client, empty_upstream_config), (downstream_client, empty_downstream_config)]
+        )
+        time.sleep(3)
+
+        # Test that CDC is disconnected - create new collection and verify it doesn't sync
+        disconnected_collection_name = self.gen_unique_name("disconnected_test")
+        upstream_client.create_collection(
+            collection_name=disconnected_collection_name,
+            schema=self.create_default_schema(upstream_client),
+        )
+
+        # Wait a reasonable time and verify it doesn't sync
+        time.sleep(10)
+        assert not downstream_client.has_collection(disconnected_collection_name), (
+            "Collection should not sync after CDC disconnection"
+        )
+
+        # Cleanup
+        self.cleanup_collection(upstream_client, test_collection_name)
+        self.cleanup_collection(upstream_client, disconnected_collection_name)

tests/python_client/cdc/testcases/test_switchover.py

@@ -0,0 +1,793 @@
+"""
+CDC sync tests for topology switchover and failover scenarios.
+"""
+
+import random
+import subprocess
+import threading
+import time
+
+from .base import TestCDCSyncBase, logger
+
+
+class TestCDCSyncSwitchover(TestCDCSyncBase):
+    """Test CDC sync behaviour during and after topology switchover / failover."""
+
+    def setup_method(self):
+        """Setup for each test method."""
+        self.resources_to_cleanup = []
+
+    def teardown_method(self):
+        """Cleanup after each test method - only cleanup upstream, downstream will sync."""
+        upstream_client = getattr(self, "_upstream_client", None)
+
+        if upstream_client:
+            for resource_type, resource_name in self.resources_to_cleanup:
+                if resource_type == "collection":
+                    self.cleanup_collection(upstream_client, resource_name)
+
+            time.sleep(1)  # Allow cleanup to sync to downstream
+
+    # -------------------------------------------------------------------------
+    # Tests
+    # -------------------------------------------------------------------------
+
+    def test_switchover_basic(
+        self,
+        upstream_client,
+        downstream_client,
+        sync_timeout,
+        switchover_helper,
+        source_cluster_id,
+        target_cluster_id,
+    ):
+        """
+        Basic switchover: insert 200 records, verify sync, switchover, insert 200 more
+        on the new source, verify count == 400 on both sides, sample verify, switch back.
+        """
+        start_time = time.time()
+        c_name = self.gen_unique_name("test_sw_basic", max_length=50)
+
+        self.log_test_start("test_switchover_basic", "SWITCHOVER_BASIC", c_name)
+        self._upstream_client = upstream_client
+        self.resources_to_cleanup.append(("collection", c_name))
+
+        try:
+            self.cleanup_collection(upstream_client, c_name)
+
+            schema = self.create_manual_id_schema(upstream_client)
+            upstream_client.create_collection(collection_name=c_name, schema=schema)
+
+            index_params = upstream_client.prepare_index_params()
+            index_params.add_index(
+                field_name="vector",
+                index_type="HNSW",
+                metric_type="L2",
+                params={"M": 16, "efConstruction": 200},
+            )
+            upstream_client.create_index(c_name, index_params)
+            upstream_client.load_collection(c_name)
+
+            # Wait for collection to appear on downstream
+            def check_create():
+                return downstream_client.has_collection(c_name)
+
+            assert self.wait_for_sync(check_create, sync_timeout, f"create collection {c_name}")
+
+            # Insert 200 records on upstream (original source)
+            batch1 = self.generate_test_data_with_id(200, start_id=0)
+            upstream_client.insert(c_name, batch1)
+            upstream_client.flush(c_name)
+
+            def check_200():
+                try:
+                    res = downstream_client.query(
+                        collection_name=c_name,
+                        filter="",
+                        output_fields=["count(*)"],
+                    )
+                    cnt = res[0]["count(*)"] if res else 0
+                    logger.info(f"[SYNC_PROGRESS] downstream count: {cnt}/200")
+                    return cnt >= 200
+                except Exception as e:
+                    logger.warning(f"Sync check failed: {e}")
+                    return False
+
+            assert self.wait_for_sync(check_200, sync_timeout, f"initial 200-record sync {c_name}")
+
+            # Switchover: target becomes new source
+            logger.info("[SWITCHOVER] Initiating basic switchover...")
+            switchover_helper(target_cluster_id, source_cluster_id)
+
+            # Insert 200 more records on the new source (previously downstream)
+            batch2 = self.generate_test_data_with_id(200, start_id=200)
+            downstream_client.insert(c_name, batch2)
+            downstream_client.flush(c_name)
+
+            def check_400_upstream():
+                try:
+                    res = upstream_client.query(
+                        collection_name=c_name,
+                        filter="",
+                        output_fields=["count(*)"],
+                    )
+                    cnt = res[0]["count(*)"] if res else 0
+                    logger.info(f"[SYNC_PROGRESS] upstream count after switchover: {cnt}/400")
+                    return cnt >= 400
+                except Exception as e:
+                    logger.warning(f"Sync check failed: {e}")
+                    return False
+
+            def check_400_downstream():
+                try:
+                    res = downstream_client.query(
+                        collection_name=c_name,
+                        filter="",
+                        output_fields=["count(*)"],
+                    )
+                    cnt = res[0]["count(*)"] if res else 0
+                    logger.info(f"[SYNC_PROGRESS] downstream count after switchover: {cnt}/400")
+                    return cnt >= 400
+                except Exception as e:
+                    logger.warning(f"Sync check failed: {e}")
+                    return False
+
+            assert self.wait_for_sync(check_400_upstream, sync_timeout, f"400-record sync upstream {c_name}")
+            assert self.wait_for_sync(check_400_downstream, sync_timeout, f"400-record sync downstream {c_name}")
+
+            # Sample verify
+            match_count, mismatch_count, details = self.verify_data_sampling(
+                downstream_client,  # new source
+                upstream_client,  # new target
+                c_name,
+                sample_ratio=0.2,
+                output_fields=["id", "vector"],
+            )
+            assert mismatch_count == 0, f"Data mismatch after basic switchover: {details[:5]}"
+
+        finally:
+            # Restore original topology
+            logger.info("[SWITCHOVER] Restoring original topology after test_switchover_basic...")
+            switchover_helper(source_cluster_id, target_cluster_id)
+            self.log_test_end("test_switchover_basic", True, time.time() - start_time)
+
+    def test_switchover_during_writes(
+        self,
+        upstream_client,
+        downstream_client,
+        sync_timeout,
+        switchover_helper,
+        source_cluster_id,
+        target_cluster_id,
+    ):
+        """
+        Switchover during concurrent writes: background thread inserts 10 batches of 100
+        with 2 s between each; switchover fires at t=12 s; join thread, flush, verify
+        both sides have the same count >= total_inserted.
+        """
+        start_time = time.time()
+        c_name = self.gen_unique_name("test_sw_writes", max_length=50)
+
+        self.log_test_start("test_switchover_during_writes", "SWITCHOVER_DURING_WRITES", c_name)
+        self._upstream_client = upstream_client
+        self.resources_to_cleanup.append(("collection", c_name))
+
+        try:
+            self.cleanup_collection(upstream_client, c_name)
+
+            schema = self.create_manual_id_schema(upstream_client)
+            upstream_client.create_collection(collection_name=c_name, schema=schema)
+
+            index_params = upstream_client.prepare_index_params()
+            index_params.add_index(
+                field_name="vector",
+                index_type="HNSW",
+                metric_type="L2",
+                params={"M": 16, "efConstruction": 200},
+            )
+            upstream_client.create_index(c_name, index_params)
+            upstream_client.load_collection(c_name)
+
+            def check_create():
+                return downstream_client.has_collection(c_name)
+
+            assert self.wait_for_sync(check_create, sync_timeout, f"create collection {c_name}")
+
+            total_inserted = 0
+            lock = threading.Lock()
+            write_error = []
+
+            def background_insert():
+                nonlocal total_inserted
+                for batch_idx in range(10):
+                    try:
+                        start_id = batch_idx * 100
+                        data = self.generate_test_data_with_id(100, start_id=start_id)
+                        # Cap pymilvus's retry loop at 10 (default 75). A batch
+                        # that lands on the old primary after it flips to
+                        # replica returns STREAMING_CODE_REPLICATE_VIOLATION;
+                        # the default retry burns ~213 s per failed batch and
+                        # makes the thread runtime unbounded when the role flip
+                        # takes longer than one batch's cadence. 10 retries
+                        # with the decorator's 3 s backoff cap bound each
+                        # failure to ~16 s.
+                        upstream_client.insert(c_name, data, retry_times=10)
+                        with lock:
+                            total_inserted += 100
+                        logger.info(f"[BACKGROUND] Inserted batch {batch_idx + 1}/10 (total: {total_inserted})")
+                    except Exception as e:
+                        logger.error(f"[BACKGROUND] Insert failed on batch {batch_idx}: {e}")
+                        write_error.append(e)
+                    time.sleep(2)
+
+            insert_thread = threading.Thread(target=background_insert, daemon=True)
+            insert_thread.start()
+
+            # Switchover after 12 s (mid-way through background inserts)
+            time.sleep(12)
+            logger.info("[SWITCHOVER] Initiating switchover during writes...")
+            switchover_helper(target_cluster_id, source_cluster_id)
+
+            # With retry_times=10 each failed batch burns ~16 s (vs ~213 s with
+            # the pymilvus default of 75). Worst case: all 10 batches fail →
+            # ~10 * (16 + 2) = 180 s. 180 s gives ~6x margin.
+            insert_thread.join(timeout=180)
+            assert not insert_thread.is_alive(), "Background insert thread did not finish in time"
+
+            # Transient STREAMING_CODE_REPLICATE_VIOLATION failures are EXPECTED
+            # during the role flip: writes that arrive on the old primary after
+            # it becomes a replica are rejected by design. The invariant that
+            # matters is that every batch which SUCCEEDED ends up replicated
+            # consistently (enforced by the up_cnt == down_cnt check below).
+            if write_error:
+                logger.warning(
+                    f"Background inserts had {len(write_error)} transient failure(s) during switchover: {write_error}"
+                )
+
+            # After switchover, downstream is the new primary. Flushing the
+            # old primary (upstream) hits the replica-role rate limiter
+            # (rate=0.1/s) and exhausts pymilvus's 75 retries. Flush the
+            # current primary instead.
+            downstream_client.flush(c_name)
+
+            expected = total_inserted
+            logger.info(f"[INFO] Total inserted: {expected}")
+
+            def check_both(client, label):
+                def _check():
+                    try:
+                        res = client.query(
+                            collection_name=c_name,
+                            filter="",
+                            output_fields=["count(*)"],
+                        )
+                        cnt = res[0]["count(*)"] if res else 0
+                        logger.info(f"[SYNC_PROGRESS] {label} count: {cnt}/{expected}")
+                        return cnt >= expected
+                    except Exception as e:
+                        logger.warning(f"Sync check failed ({label}): {e}")
+                        return False
+
+                return _check
+
+            assert self.wait_for_sync(
+                check_both(upstream_client, "upstream"),
+                sync_timeout,
+                f"upstream count>={expected} after switchover-during-writes",
+            )
+            assert self.wait_for_sync(
+                check_both(downstream_client, "downstream"),
+                sync_timeout,
+                f"downstream count>={expected} after switchover-during-writes",
+            )
+
+            up_res = upstream_client.query(collection_name=c_name, filter="", output_fields=["count(*)"])
+            down_res = downstream_client.query(collection_name=c_name, filter="", output_fields=["count(*)"])
+            up_cnt = up_res[0]["count(*)"] if up_res else 0
+            down_cnt = down_res[0]["count(*)"] if down_res else 0
+            assert up_cnt == down_cnt, (
+                f"Count mismatch after switchover-during-writes: upstream={up_cnt}, downstream={down_cnt}"
+            )
+
+        finally:
+            logger.info("[SWITCHOVER] Restoring original topology after test_switchover_during_writes...")
+            switchover_helper(source_cluster_id, target_cluster_id)
+            self.log_test_end("test_switchover_during_writes", True, time.time() - start_time)
+
+    def test_switchover_with_all_data_types(
+        self,
+        upstream_client,
+        downstream_client,
+        sync_timeout,
+        switchover_helper,
+        source_cluster_id,
+        target_cluster_id,
+    ):
+        """
+        Switchover with comprehensive data types: insert 100 records, verify sync,
+        switchover, verify scalar field sampling, switch back.
+        """
+        start_time = time.time()
+        c_name = self.gen_unique_name("test_sw_dtypes", max_length=50)
+
+        self.log_test_start(
+            "test_switchover_with_all_data_types",
+            "SWITCHOVER_ALL_DTYPES",
+            c_name,
+        )
+        self._upstream_client = upstream_client
+        self.resources_to_cleanup.append(("collection", c_name))
+
+        try:
+            self.cleanup_collection(upstream_client, c_name)
+
+            schema = self.create_comprehensive_schema(upstream_client)
+            upstream_client.create_collection(collection_name=c_name, schema=schema)
+
+            # Index every vector field; load_collection fails otherwise.
+            index_params = upstream_client.prepare_index_params()
+            index_params.add_index(field_name="float_vector", index_type="AUTOINDEX", metric_type="L2")
+            index_params.add_index(field_name="float16_vector", index_type="AUTOINDEX", metric_type="L2")
+            index_params.add_index(field_name="binary_vector", index_type="BIN_FLAT", metric_type="HAMMING")
+            index_params.add_index(field_name="sparse_vector", index_type="SPARSE_INVERTED_INDEX", metric_type="IP")
+            upstream_client.create_index(c_name, index_params)
+            upstream_client.load_collection(c_name)
+
+            def check_create():
+                return downstream_client.has_collection(c_name)
+
+            assert self.wait_for_sync(check_create, sync_timeout, f"create collection {c_name}")
+
+            data = self.generate_comprehensive_test_data(100)
+            upstream_client.insert(c_name, data)
+            upstream_client.flush(c_name)
+
+            def check_100():
+                try:
+                    res = downstream_client.query(
+                        collection_name=c_name,
+                        filter="",
+                        output_fields=["count(*)"],
+                    )
+                    cnt = res[0]["count(*)"] if res else 0
+                    logger.info(f"[SYNC_PROGRESS] downstream count: {cnt}/100")
+                    return cnt >= 100
+                except Exception as e:
+                    logger.warning(f"Sync check failed: {e}")
+                    return False
+
+            assert self.wait_for_sync(check_100, sync_timeout, f"100-record sync {c_name}")
+
+            # Switchover
+            logger.info("[SWITCHOVER] Initiating switchover with all data types...")
+            switchover_helper(target_cluster_id, source_cluster_id)
+
+            # After switchover, verify scalar field sampling (new source is downstream)
+            scalar_fields = [
+                "bool_field",
+                "int8_field",
+                "int16_field",
+                "int32_field",
+                "int64_field",
+                "float_field",
+                "double_field",
+                "varchar_field",
+            ]
+            match_count, mismatch_count, details = self.verify_data_sampling(
+                downstream_client,  # new source
+                upstream_client,  # new target
+                c_name,
+                sample_ratio=0.3,
+                output_fields=scalar_fields,
+            )
+            logger.info(f"[RESULT] All-dtypes sampling — match={match_count}, mismatch={mismatch_count}")
+            assert mismatch_count == 0, f"Data mismatch after switchover with all types: {details[:5]}"
+
+        finally:
+            logger.info("[SWITCHOVER] Restoring original topology after test_switchover_with_all_data_types...")
+            switchover_helper(source_cluster_id, target_cluster_id)
+            self.log_test_end(
+                "test_switchover_with_all_data_types",
+                True,
+                time.time() - start_time,
+            )
+
+    def test_switchover_with_loaded_collection(
+        self,
+        upstream_client,
+        downstream_client,
+        sync_timeout,
+        switchover_helper,
+        source_cluster_id,
+        target_cluster_id,
+    ):
+        """
+        Switchover with a loaded collection: verify search works on downstream before
+        switchover; after switchover verify search works on both sides.
+        """
+        start_time = time.time()
+        c_name = self.gen_unique_name("test_sw_loaded", max_length=50)
+
+        self.log_test_start(
+            "test_switchover_with_loaded_collection",
+            "SWITCHOVER_LOADED",
+            c_name,
+        )
+        self._upstream_client = upstream_client
+        self.resources_to_cleanup.append(("collection", c_name))
+
+        try:
+            self.cleanup_collection(upstream_client, c_name)
+
+            schema = self.create_default_schema(upstream_client)
+            upstream_client.create_collection(collection_name=c_name, schema=schema)
+
+            index_params = upstream_client.prepare_index_params()
+            index_params.add_index(
+                field_name="vector",
+                index_type="HNSW",
+                metric_type="L2",
+                params={"M": 16, "efConstruction": 200},
+            )
+            upstream_client.create_index(c_name, index_params)
+            upstream_client.load_collection(c_name)
+
+            def check_create():
+                return downstream_client.has_collection(c_name)
+
+            assert self.wait_for_sync(check_create, sync_timeout, f"create collection {c_name}")
+
+            data = self.generate_test_data(200)
+            upstream_client.insert(c_name, data)
+            upstream_client.flush(c_name)
+
+            def check_200():
+                try:
+                    res = downstream_client.query(
+                        collection_name=c_name,
+                        filter="",
+                        output_fields=["count(*)"],
+                    )
+                    cnt = res[0]["count(*)"] if res else 0
+                    logger.info(f"[SYNC_PROGRESS] downstream count: {cnt}/200")
+                    return cnt >= 200
+                except Exception as e:
+                    logger.warning(f"Sync check failed: {e}")
+                    return False
+
+            assert self.wait_for_sync(check_200, sync_timeout, f"200-record sync {c_name}")
+
+            # Verify search works on downstream before switchover
+            q_vec = [random.random() for _ in range(128)]
+            pre_down_results = downstream_client.search(
+                collection_name=c_name,
+                data=[q_vec],
+                anns_field="vector",
+                search_params={"metric_type": "L2"},
+                limit=5,
+                output_fields=["id"],
+            )
+            assert pre_down_results and len(pre_down_results[0]) > 0, (
+                "Search on downstream returned no results before switchover"
+            )
+
+            # Switchover
+            logger.info("[SWITCHOVER] Initiating switchover with loaded collection...")
+            switchover_helper(target_cluster_id, source_cluster_id)
+
+            # Verify search still works on both sides after switchover
+            for client, label in [(upstream_client, "upstream"), (downstream_client, "downstream")]:
+                results = client.search(
+                    collection_name=c_name,
+                    data=[q_vec],
+                    anns_field="vector",
+                    search_params={"metric_type": "L2"},
+                    limit=5,
+                    output_fields=["id"],
+                )
+                assert results and len(results[0]) > 0, f"Search returned no results on {label} after switchover"
+                logger.info(f"[VERIFY] Search on {label} after switchover returned {len(results[0])} results — OK")
+
+        finally:
+            logger.info("[SWITCHOVER] Restoring original topology after test_switchover_with_loaded_collection...")
+            switchover_helper(source_cluster_id, target_cluster_id)
+            self.log_test_end(
+                "test_switchover_with_loaded_collection",
+                True,
+                time.time() - start_time,
+            )
+
+    def test_switchover_with_index(
+        self,
+        upstream_client,
+        downstream_client,
+        sync_timeout,
+        switchover_helper,
+        source_cluster_id,
+        target_cluster_id,
+    ):
+        """
+        Switchover after index creation: wait for index to sync, switchover, verify
+        list_indexes returns the same index on both sides.
+        """
+        start_time = time.time()
+        c_name = self.gen_unique_name("test_sw_index", max_length=50)
+
+        self.log_test_start("test_switchover_with_index", "SWITCHOVER_INDEX", c_name)
+        self._upstream_client = upstream_client
+        self.resources_to_cleanup.append(("collection", c_name))
+
+        try:
+            self.cleanup_collection(upstream_client, c_name)
+
+            schema = self.create_default_schema(upstream_client)
+            upstream_client.create_collection(collection_name=c_name, schema=schema)
+
+            # Insert data first
+            data = self.generate_test_data(200)
+            upstream_client.insert(c_name, data)
+            upstream_client.flush(c_name)
+
+            # Create HNSW index
+            index_params = upstream_client.prepare_index_params()
+            index_params.add_index(
+                field_name="vector",
+                index_type="HNSW",
+                metric_type="COSINE",
+                params={"M": 16, "efConstruction": 200},
+            )
+            upstream_client.create_index(c_name, index_params)
+
+            # Wait for index to sync to downstream
+            def check_index_sync():
+                try:
+                    indexes = downstream_client.list_indexes(c_name)
+                    result = len(indexes) > 0
+                    logger.info(f"[SYNC_PROGRESS] downstream indexes: {indexes} (has_index={result})")
+                    return result
+                except Exception as e:
+                    logger.warning(f"Index sync check failed: {e}")
+                    return False
+
+            assert self.wait_for_sync(check_index_sync, sync_timeout, f"index sync {c_name}"), (
+                f"Index did not sync to downstream within {sync_timeout}s"
+            )
+
+            # Switchover
+            logger.info("[SWITCHOVER] Initiating switchover with index...")
+            switchover_helper(target_cluster_id, source_cluster_id)
+
+            # Verify list_indexes on both sides
+            up_indexes = upstream_client.list_indexes(c_name)
+            down_indexes = downstream_client.list_indexes(c_name)
+
+            logger.info(f"[VERIFY] After switchover — upstream indexes={up_indexes}, downstream indexes={down_indexes}")
+            assert len(up_indexes) > 0, "No indexes on upstream after switchover"
+            assert len(down_indexes) > 0, "No indexes on downstream after switchover"
+            assert set(up_indexes) == set(down_indexes), (
+                f"Index lists differ after switchover: up={up_indexes}, down={down_indexes}"
+            )
+
+        finally:
+            logger.info("[SWITCHOVER] Restoring original topology after test_switchover_with_index...")
+            switchover_helper(source_cluster_id, target_cluster_id)
+            self.log_test_end("test_switchover_with_index", True, time.time() - start_time)
+
+    def test_rapid_switchover_stress(
+        self,
+        upstream_client,
+        downstream_client,
+        sync_timeout,
+        switchover_helper,
+        source_cluster_id,
+        target_cluster_id,
+    ):
+        """
+        Rapid-switchover stress: loop 5 times — write 50 records to current source,
+        switchover. After loop, wait 30 s and verify counts match. Restore original topology.
+        """
+        start_time = time.time()
+        c_name = self.gen_unique_name("test_sw_stress", max_length=50)
+
+        self.log_test_start("test_rapid_switchover_stress", "RAPID_SWITCHOVER_STRESS", c_name)
+        self._upstream_client = upstream_client
+        self.resources_to_cleanup.append(("collection", c_name))
+
+        try:
+            self.cleanup_collection(upstream_client, c_name)
+
+            schema = self.create_manual_id_schema(upstream_client)
+            upstream_client.create_collection(collection_name=c_name, schema=schema)
+
+            index_params = upstream_client.prepare_index_params()
+            index_params.add_index(
+                field_name="vector",
+                index_type="HNSW",
+                metric_type="COSINE",
+                params={"M": 16, "efConstruction": 200},
+            )
+            upstream_client.create_index(c_name, index_params)
+            upstream_client.load_collection(c_name)
+
+            def check_create():
+                return downstream_client.has_collection(c_name)
+
+            assert self.wait_for_sync(check_create, sync_timeout, f"create collection {c_name}")
+
+            # Track topology state
+            current_source_id = source_cluster_id
+            current_target_id = target_cluster_id
+            current_source_client = upstream_client
+            total_inserted = 0
+
+            for iteration in range(5):
+                # Write 50 records to current source
+                start_id = iteration * 50
+                data = self.generate_test_data_with_id(50, start_id=start_id)
+                current_source_client.insert(c_name, data)
+                current_source_client.flush(c_name)
+                total_inserted += 50
+
+                logger.info(
+                    f"[STRESS] Iteration {iteration + 1}/5: inserted 50 records "
+                    f"(total={total_inserted}), switching over..."
+                )
+
+                # Switchover
+                switchover_helper(current_target_id, current_source_id)
+
+                # Swap topology
+                current_source_id, current_target_id = current_target_id, current_source_id
+                current_source_client = downstream_client if current_source_id == target_cluster_id else upstream_client
+
+            logger.info(
+                f"[STRESS] All 5 switchovers done. Waiting 30s for final sync (total_inserted={total_inserted})..."
+            )
+            time.sleep(30)
+
+            # Verify counts match on both sides
+            up_res = upstream_client.query(collection_name=c_name, filter="", output_fields=["count(*)"])
+            down_res = downstream_client.query(collection_name=c_name, filter="", output_fields=["count(*)"])
+            up_cnt = up_res[0]["count(*)"] if up_res else 0
+            down_cnt = down_res[0]["count(*)"] if down_res else 0
+
+            logger.info(f"[VERIFY] After stress — upstream={up_cnt}, downstream={down_cnt}, expected>={total_inserted}")
+            assert up_cnt >= total_inserted, f"Upstream count {up_cnt} < expected {total_inserted} after stress"
+            assert down_cnt >= total_inserted, f"Downstream count {down_cnt} < expected {total_inserted} after stress"
+            assert up_cnt == down_cnt, f"Count mismatch after stress: upstream={up_cnt}, downstream={down_cnt}"
+
+        finally:
+            # Restore to original topology
+            logger.info("[SWITCHOVER] Restoring original topology after test_rapid_switchover_stress...")
+            switchover_helper(source_cluster_id, target_cluster_id)
+            self.log_test_end("test_rapid_switchover_stress", True, time.time() - start_time)
+
+    def test_failover_source_down(
+        self,
+        upstream_client,
+        downstream_client,
+        sync_timeout,
+        switchover_helper,
+        source_cluster_id,
+        target_cluster_id,
+        milvus_ns,
+    ):
+        """
+        Failover when source goes down: insert 500 records, verify sync, kill source pods,
+        verify target count >= 500, wait for source to recover and verify count >= 500.
+        """
+        start_time = time.time()
+        c_name = self.gen_unique_name("test_failover_src", max_length=50)
+
+        self.log_test_start("test_failover_source_down", "FAILOVER_SOURCE_DOWN", c_name)
+        self._upstream_client = upstream_client
+        self.resources_to_cleanup.append(("collection", c_name))
+
+        try:
+            self.cleanup_collection(upstream_client, c_name)
+
+            schema = self.create_manual_id_schema(upstream_client)
+            upstream_client.create_collection(collection_name=c_name, schema=schema)
+
+            index_params = upstream_client.prepare_index_params()
+            index_params.add_index(
+                field_name="vector",
+                index_type="HNSW",
+                metric_type="L2",
+                params={"M": 16, "efConstruction": 200},
+            )
+            upstream_client.create_index(c_name, index_params)
+            upstream_client.load_collection(c_name)
+
+            def check_create():
+                return downstream_client.has_collection(c_name)
+
+            assert self.wait_for_sync(check_create, sync_timeout, f"create collection {c_name}")
+
+            # Insert 500 records
+            data = self.generate_test_data_with_id(500, start_id=0)
+            upstream_client.insert(c_name, data)
+            upstream_client.flush(c_name)
+
+            def check_500_downstream():
+                try:
+                    res = downstream_client.query(
+                        collection_name=c_name,
+                        filter="",
+                        output_fields=["count(*)"],
+                    )
+                    cnt = res[0]["count(*)"] if res else 0
+                    logger.info(f"[SYNC_PROGRESS] downstream count: {cnt}/500")
+                    return cnt >= 500
+                except Exception as e:
+                    logger.warning(f"Sync check failed: {e}")
+                    return False
+
+            assert self.wait_for_sync(check_500_downstream, sync_timeout, f"500-record sync {c_name}"), (
+                f"Downstream did not receive 500 records within {sync_timeout}s"
+            )
+
+            # Kill source pods forcefully
+            logger.info(f"[FAILOVER] Killing source pods (instance={source_cluster_id}, ns={milvus_ns})...")
+            kill_cmd = [
+                "kubectl",
+                "delete",
+                "pods",
+                "-l",
+                f"app.kubernetes.io/instance={source_cluster_id}",
+                "-n",
+                milvus_ns,
+                "--grace-period=0",
+                "--force",
+            ]
+            result = subprocess.run(kill_cmd, capture_output=True, text=True)
+            logger.info(
+                f"[FAILOVER] kubectl output: stdout={result.stdout!r}, stderr={result.stderr!r}, rc={result.returncode}"
+            )
+
+            # Wait 60 s and verify target still has data
+            logger.info("[FAILOVER] Waiting 60s after pod kill...")
+            time.sleep(60)
+
+            def check_target_intact():
+                try:
+                    res = downstream_client.query(
+                        collection_name=c_name,
+                        filter="",
+                        output_fields=["count(*)"],
+                    )
+                    cnt = res[0]["count(*)"] if res else 0
+                    logger.info(f"[FAILOVER] Target count after kill: {cnt}")
+                    return cnt >= 500
+                except Exception as e:
+                    logger.warning(f"Target check failed: {e}")
+                    return False
+
+            assert self.wait_for_sync(check_target_intact, 30, f"target intact after source kill {c_name}"), (
+                "Target lost data after source pod kill"
+            )
+
+            # Wait 120 s more for source to recover, then verify
+            logger.info("[FAILOVER] Waiting 120s for source recovery...")
+            time.sleep(120)
+
+            def check_source_recovered():
+                try:
+                    res = upstream_client.query(
+                        collection_name=c_name,
+                        filter="",
+                        output_fields=["count(*)"],
+                    )
+                    cnt = res[0]["count(*)"] if res else 0
+                    logger.info(f"[FAILOVER] Source count after recovery: {cnt}")
+                    return cnt >= 500
+                except Exception as e:
+                    logger.warning(f"Source recovery check failed: {e}")
+                    return False
+
+            assert self.wait_for_sync(check_source_recovered, 60, f"source recovery {c_name}"), (
+                "Source did not recover with expected data count within timeout"
+            )
+
+        finally:
+            self.log_test_end("test_failover_source_down", True, time.time() - start_time)

tests/python_client/chaos/README.md

@@ -0,0 +1,95 @@
+# Chaos Tests
+## Goal
+Chaos tests are designed to check the reliability of Milvus.
+
+For instance, if one pod is killed:
+   - verify that it restarts automatically 
+   - verify that the related operation fails, while the other operations keep working successfully during the absence of the pod
+   - verify that all the operations work successfully after the pod back to running state
+   - verify that no data lost
+
+## Prerequisite
+Chaos tests run in pytest framework, same as e2e tests. 
+
+Please refer to [Run E2E Tests](https://github.com/milvus-io/milvus/blob/master/tests/README.md)
+
+## Flow Chart
+
+<img src="../graphs/chaos_test_flow_chart.jpg" alt="Chaos Test Flow Chart" width=50%/>
+
+## Test Scenarios
+### Milvus in cluster mode
+#### pod kill
+
+Kill pod every 5s
+
+#### pod network partition
+
+Two direction(to and from) network isolation between a pod and the rest of the pods
+
+#### pod failure
+
+Set the pod（querynode, indexnode and datanode）as multiple replicas, make one of them failure, and test milvus's functionality
+
+#### pod memory stress
+
+Limit the memory resource of pod and generate plenty of stresses over a group of pods
+
+### Milvus in standalone mode
+1. standalone pod is killed
+
+2. minio pod is killed
+
+## How it works
+- Test scenarios are designed by different chaos objects
+- Every chaos object is defined in one yaml file locates in  folder `chaos_objects`
+- Every chaos yaml file specified by `ALL_CHAOS_YAMLS` in `constants.py` would be parsed as a parameter and be passed into `test_chaos.py`
+- All expectations of every scenario are defined in `testcases.yaml` locates in folder `chaos_objects`
+- [Chaos Mesh](https://chaos-mesh.org/) is used to inject chaos into Milvus in `test_chaos.py`
+
+## Run
+### Manually
+Run a single test scenario manually(take query node pod is killed as instance):
+1. update `ALL_CHAOS_YAMLS = 'chaos_querynode_podkill.yaml'` in `constants.py`
+
+2. run the commands below:
+   ```bash
+   cd /milvus/tests/python_client/chaos
+
+   pytest test_chaos.py --host ${Milvus_IP} -v
+   ```
+Run multiple test scenario in a category manually(take network partition chaos for all pods as instance):
+
+1. update `ALL_CHAOS_YAMLS = 'chaos_*_network_partition.yaml'` in `constants.py`
+
+2. run the commands below:
+   ```bash
+   cd /milvus/tests/python_client/chaos
+
+   pytest test_chaos.py --host ${Milvus_IP} -v
+   ```
+### Automation Scripts
+Run test scenario automatically:
+1. update chaos type and pod in `chaos_test.sh`
+2. run the commands below:
+   ```bash
+   cd /milvus/tests/python_client/chaos
+   # in this step, script will install milvus with replicas_num and run testcase
+   bash chaos_test.sh ${pod} ${chaos_type} ${chaos_task} ${replicas_num}
+   # example: bash chaos_test.sh querynode pod_kill chaos-test 2
+   ```
+### Nightly 
+still in planning 
+
+### Todo
+- [ ] network attack
+- [ ] clock skew
+- [ ] IO injection
+
+## How to contribute
+* Get familiar with chaos engineering and [Chaos Mesh](https://chaos-mesh.org)
+* Design chaos scenarios, preferring to pick from todo list
+* Generate yaml file for your chaos scenarios. You can create a chaos experiment in chaos-dashboard, then download the yaml file of it.
+* Add yaml file to chaos_objects dir and rename it as `chaos_${component_name}_${chaos_type}.yaml`. Make sure `kubectl apply -f ${your_chaos_yaml_file}` can take effect
+* Add testcase in `testcases.yaml`. You should figure out the expectation of milvus during the chaos
+* Run your added testcase according to `Manually` above and check whether it as your expectation

tests/python_client/chaos/chaos_commons.py

@@ -0,0 +1,137 @@
+import glob
+import os
+import threading
+import time
+
+import pytest
+from chaos import constants
+from utils.util_log import test_log as log
+from yaml import full_load
+
+
+def check_config(chaos_config):
+    if not chaos_config.get("kind", None):
+        raise Exception("kind must be specified")
+    if not chaos_config.get("spec", None):
+        raise Exception("spec must be specified")
+    if "action" not in chaos_config.get("spec", None):
+        raise Exception("action must be specified in spec")
+    if "selector" not in chaos_config.get("spec", None):
+        raise Exception("selector must be specified in spec")
+    return True
+
+
+def reset_counting(checkers={}):
+    """reset checker counts for all checker threads"""
+    for ch in checkers.values():
+        ch.reset()
+
+
+def gen_experiment_config(yaml):
+    """load the yaml file of chaos experiment"""
+    with open(yaml) as f:
+        _config = full_load(f)
+        f.close()
+    return _config
+
+
+def start_monitor_threads(checkers={}):
+    """start the threads by checkers"""
+    tasks = []
+    for k, ch in checkers.items():
+        ch._keep_running = True
+        t = threading.Thread(target=ch.keep_running, args=(), name=k, daemon=True)
+        t.start()
+        tasks.append(t)
+    return tasks
+
+
+def check_thread_status(tasks):
+    """check the status of all threads"""
+    for t in tasks:
+        if t.is_alive():
+            log.info(f"thread {t.name} is still running")
+        else:
+            log.info(f"thread {t.name} is not running")
+
+
+def get_env_variable_by_name(name):
+    """get env variable by name"""
+    try:
+        env_var = os.environ[name]
+        log.debug(f"env_variable: {env_var}")
+        return str(env_var)
+    except Exception as e:
+        log.debug(f"fail to get env variables, error: {str(e)}")
+        return None
+
+
+def get_chaos_yamls():
+    """get chaos yaml file(s) from configured environment path"""
+    chaos_env = get_env_variable_by_name(constants.CHAOS_CONFIG_ENV)
+    if chaos_env is not None:
+        if os.path.isdir(chaos_env):
+            log.debug(f"chaos_env is a dir: {chaos_env}")
+            return glob.glob(chaos_env + "chaos_*.yaml")
+        elif os.path.isfile(chaos_env):
+            log.debug(f"chaos_env is a file: {chaos_env}")
+            return [chaos_env]
+        else:
+            # not a valid directory, return default
+            pass
+    log.debug("not a valid directory or file, return default chaos config path")
+    return glob.glob(constants.TESTS_CONFIG_LOCATION + constants.ALL_CHAOS_YAMLS)
+
+
+def reconnect(connections, alias="default", timeout=360):
+    """trying to connect by connection alias"""
+    is_connected = False
+    start = time.time()
+    end = time.time()
+    while not is_connected or end - start < timeout:
+        try:
+            connections.connect(alias)
+            is_connected = True
+        except Exception as e:
+            log.debug(f"fail to connect, error: {str(e)}")
+            time.sleep(10)
+        end = time.time()
+    else:
+        log.info(f"failed to reconnect after {timeout} seconds")
+    return connections.connect(alias)
+
+
+def assert_statistic(checkers, expectations={}, succ_rate_threshold=0.95, fail_rate_threshold=0.49):
+    for k in checkers.keys():
+        # expect succ if no expectations
+        succ_rate = checkers[k].succ_rate()
+        total = checkers[k].total()
+        average_time = checkers[k].average_time
+        error_messages = getattr(checkers[k], "error_messages", set())
+        if expectations.get(k, "") == constants.FAIL:
+            log.info(f"Expect Fail: {str(k)} succ rate {succ_rate}, total: {total}, average time: {average_time:.4f}")
+            pytest.assume(
+                succ_rate < fail_rate_threshold or total < 2,
+                f"Expect Fail: {str(k)} succ rate {succ_rate}, total: {total}, average time: {average_time:.4f}",
+            )
+        else:
+            log.info(f"Expect Succ: {str(k)} succ rate {succ_rate}, total: {total}, average time: {average_time:.4f}")
+            # Build assertion message with error details
+            assert_msg = (
+                f"Expect Succ: {str(k)} succ rate {succ_rate}, total: {total}, average time: {average_time:.4f}"
+            )
+            if error_messages:
+                error_details = "; ".join(error_messages)
+                assert_msg += f", unique errors({len(error_messages)}): [{error_details}]"
+            if total == 0:
+                # Checker never completed any operation. This indicates either the checker
+                # thread failed to start or the service was completely unavailable. Treat
+                # it as a real failure rather than silently skipping.
+                pytest.assume(False, f"{str(k)} never completed any operation (total=0)")
+            else:
+                # total >= 1: at least one operation completed, succ_rate is meaningful.
+                # We no longer require total > 2 because low-frequency checkers (e.g.
+                # AddFieldChecker, AddVectorFieldChecker) legitimately complete only a few
+                # operations within a 10-minute window, and succ_rate=1.0 with total=2 is
+                # a valid passing result.
+                pytest.assume(succ_rate >= succ_rate_threshold, assert_msg)

tests/python_client/chaos/chaos_objects/container_kill/chaos_datacoord_container_kill.yaml

@@ -0,0 +1,24 @@
+kind: Schedule
+apiVersion: chaos-mesh.org/v1alpha1
+metadata:
+  name: test-datacoord-container-kill
+  namespace: chaos-testing
+spec:
+  schedule: '*/5 * * * * *'
+  startingDeadlineSeconds: 60
+  concurrencyPolicy: Forbid
+  historyLimit: 1
+  type: PodChaos
+  podChaos:
+    selector:
+      namespaces:
+        - chaos-testing
+      labelSelectors:
+        app.kubernetes.io/instance: milvus-chaos
+        app.kubernetes.io/name: milvus
+        component: datacoord
+    mode: fixed
+    value: "1"
+    action: container-kill
+    containerNames:
+      - datacoord

tests/python_client/chaos/chaos_objects/container_kill/chaos_datanode_container_kill.yaml

@@ -0,0 +1,24 @@
+kind: Schedule
+apiVersion: chaos-mesh.org/v1alpha1
+metadata:
+  name: test-datanode-container-kill
+  namespace: chaos-testing
+spec:
+  schedule: '*/5 * * * * *'
+  startingDeadlineSeconds: 60
+  concurrencyPolicy: Forbid
+  historyLimit: 1
+  type: PodChaos
+  podChaos:
+    selector:
+      namespaces:
+        - chaos-testing
+      labelSelectors:
+        app.kubernetes.io/instance: milvus-chaos
+        app.kubernetes.io/name: milvus
+        component: datanode
+    mode: fixed
+    value: "1"
+    action: container-kill
+    containerNames:
+      - datanode

tests/python_client/chaos/chaos_objects/container_kill/chaos_etcd_container_kill.yaml

@@ -0,0 +1,23 @@
+kind: Schedule
+apiVersion: chaos-mesh.org/v1alpha1
+metadata:
+  name: test-etcd-container-kill
+  namespace: chaos-testing
+spec:
+  schedule: '*/5 * * * * *'
+  startingDeadlineSeconds: 60
+  concurrencyPolicy: Forbid
+  historyLimit: 1
+  type: PodChaos
+  podChaos:
+    selector:
+      namespaces:
+        - chaos-testing
+      labelSelectors:
+        app.kubernetes.io/instance: milvus-chaos
+        app.kubernetes.io/name: etcd
+    mode: fixed
+    value: "1"
+    action: container-kill
+    containerNames:
+      - etcd

tests/python_client/chaos/chaos_objects/container_kill/chaos_indexcoord_container_kill.yaml

@@ -0,0 +1,24 @@
+kind: Schedule
+apiVersion: chaos-mesh.org/v1alpha1
+metadata:
+  name: test-indexcoord-container-kill
+  namespace: chaos-testing
+spec:
+  schedule: '*/5 * * * * *'
+  startingDeadlineSeconds: 60
+  concurrencyPolicy: Forbid
+  historyLimit: 1
+  type: PodChaos
+  podChaos:
+    selector:
+      namespaces:
+        - chaos-testing
+      labelSelectors:
+        app.kubernetes.io/instance: milvus-chaos
+        app.kubernetes.io/name: milvus
+        component: indexcoord
+    mode: fixed
+    value: "1"
+    action: container-kill
+    containerNames:
+      - indexcoord

tests/python_client/chaos/chaos_objects/container_kill/chaos_indexnode_container_kill.yaml

@@ -0,0 +1,24 @@
+kind: Schedule
+apiVersion: chaos-mesh.org/v1alpha1
+metadata:
+  name: test-indexnode-container-kill
+  namespace: chaos-testing
+spec:
+  schedule: '*/5 * * * * *'
+  startingDeadlineSeconds: 60
+  concurrencyPolicy: Forbid
+  historyLimit: 1
+  type: PodChaos
+  podChaos:
+    selector:
+      namespaces:
+        - chaos-testing
+      labelSelectors:
+        app.kubernetes.io/instance: milvus-chaos
+        app.kubernetes.io/name: milvus
+        component: indexnode
+    mode: fixed
+    value: "1"
+    action: container-kill
+    containerNames:
+      - indexnode

tests/python_client/chaos/chaos_objects/container_kill/chaos_kafka_container_kill.yaml

@@ -0,0 +1,24 @@
+kind: Schedule
+apiVersion: chaos-mesh.org/v1alpha1
+metadata:
+  name: test-kafka-container-kill
+  namespace: chaos-testing
+spec:
+  schedule: '*/5 * * * * *'
+  startingDeadlineSeconds: 60
+  concurrencyPolicy: Forbid
+  historyLimit: 1
+  type: PodChaos
+  podChaos:
+    selector:
+      namespaces:
+        - chaos-testing
+      labelSelectors:
+        app.kubernetes.io/instance: milvus-chaos
+        app.kubernetes.io/name: kafka
+        app.kubernetes.io/component: kafka
+    mode: fixed
+    value: "1"
+    action: container-kill
+    containerNames:
+      - kafka

tests/python_client/chaos/chaos_objects/container_kill/chaos_minio_container_kill.yaml

@@ -0,0 +1,23 @@
+kind: Schedule
+apiVersion: chaos-mesh.org/v1alpha1
+metadata:
+  name: test-minio-container-kill
+  namespace: chaos-testing
+spec:
+  schedule: '*/5 * * * * *'
+  startingDeadlineSeconds: 60
+  concurrencyPolicy: Forbid
+  historyLimit: 1
+  type: PodChaos
+  podChaos:
+    selector:
+      namespaces:
+        - chaos-testing
+      labelSelectors:
+        release: milvus-chaos
+        app: minio
+    mode: fixed
+    value: "1"
+    action: container-kill
+    containerNames:
+      - minio

tests/python_client/chaos/chaos_objects/container_kill/chaos_mixcoord_container_kill.yaml

@@ -0,0 +1,24 @@
+kind: Schedule
+apiVersion: chaos-mesh.org/v1alpha1
+metadata:
+  name: test-mixcoord-container-kill
+  namespace: chaos-testing
+spec:
+  schedule: '*/5 * * * * *'
+  startingDeadlineSeconds: 60
+  concurrencyPolicy: Forbid
+  historyLimit: 1
+  type: PodChaos
+  podChaos:
+    selector:
+      namespaces:
+        - chaos-testing
+      labelSelectors:
+        app.kubernetes.io/instance: milvus-chaos
+        app.kubernetes.io/name: milvus
+        component: mixcoord
+    mode: fixed
+    value: "1"
+    action: container-kill
+    containerNames:
+      - mixcoord

tests/python_client/chaos/chaos_objects/container_kill/chaos_proxy_container_kill.yaml

@@ -0,0 +1,24 @@
+kind: Schedule
+apiVersion: chaos-mesh.org/v1alpha1
+metadata:
+  name: test-proxy-container-kill
+  namespace: chaos-testing
+spec:
+  schedule: '*/5 * * * * *'
+  startingDeadlineSeconds: 60
+  concurrencyPolicy: Forbid
+  historyLimit: 1
+  type: PodChaos
+  podChaos:
+    selector:
+      namespaces:
+        - chaos-testing
+      labelSelectors:
+        app.kubernetes.io/instance: milvus-chaos
+        app.kubernetes.io/name: milvus
+        component: proxy
+    mode: fixed
+    value: "1"
+    action: container-kill
+    containerNames:
+      - proxy

tests/python_client/chaos/chaos_objects/container_kill/chaos_pulsar_container_kill.yaml

@@ -0,0 +1,21 @@
+kind: Schedule
+apiVersion: chaos-mesh.org/v1alpha1
+metadata:
+  name: test-pulsar-container-kill
+  namespace: chaos-testing
+spec:
+  schedule: '*/5 * * * * *'
+  startingDeadlineSeconds: 60
+  concurrencyPolicy: Forbid
+  historyLimit: 1
+  type: PodChaos
+  podChaos:
+    selector:
+      namespaces:
+        - chaos-testing
+      labelSelectors:
+        release: milvus-chaos
+        app: pulsarv3
+    mode: fixed
+    value: "1"
+    action: container-kill

tests/python_client/chaos/chaos_objects/container_kill/chaos_querycoord_container_kill.yaml

@@ -0,0 +1,24 @@
+kind: Schedule
+apiVersion: chaos-mesh.org/v1alpha1
+metadata:
+  name: test-querycoord-container-kill
+  namespace: chaos-testing
+spec:
+  schedule: '*/5 * * * * *'
+  startingDeadlineSeconds: 60
+  concurrencyPolicy: Forbid
+  historyLimit: 1
+  type: PodChaos
+  podChaos:
+    selector:
+      namespaces:
+        - chaos-testing
+      labelSelectors:
+        app.kubernetes.io/instance: milvus-chaos
+        app.kubernetes.io/name: milvus
+        component: querycoord
+    mode: fixed
+    value: "1"
+    action: container-kill
+    containerNames:
+      - querycoord

tests/python_client/chaos/chaos_objects/container_kill/chaos_querynode_container_kill.yaml

@@ -0,0 +1,24 @@
+kind: Schedule
+apiVersion: chaos-mesh.org/v1alpha1
+metadata:
+  name: test-querynode-container-kill
+  namespace: chaos-testing
+spec:
+  schedule: '*/5 * * * * *'
+  startingDeadlineSeconds: 60
+  concurrencyPolicy: Forbid
+  historyLimit: 1
+  type: PodChaos
+  podChaos:
+    selector:
+      namespaces:
+        - chaos-testing
+      labelSelectors:
+        app.kubernetes.io/instance: milvus-chaos
+        app.kubernetes.io/name: milvus
+        component: querynode
+    mode: fixed
+    value: "1"
+    action: container-kill
+    containerNames:
+      - querynode

tests/python_client/chaos/chaos_objects/container_kill/chaos_rootcoord_container_kill.yaml

@@ -0,0 +1,24 @@
+kind: Schedule
+apiVersion: chaos-mesh.org/v1alpha1
+metadata:
+  name: test-rootcoord-container-kill
+  namespace: chaos-testing
+spec:
+  schedule: '*/5 * * * * *'
+  startingDeadlineSeconds: 60
+  concurrencyPolicy: Forbid
+  historyLimit: 1
+  type: PodChaos
+  podChaos:
+    selector:
+      namespaces:
+        - chaos-testing
+      labelSelectors:
+        app.kubernetes.io/instance: milvus-chaos
+        app.kubernetes.io/name: milvus
+        component: rootcoord
+    mode: fixed
+    value: "1"
+    action: container-kill
+    containerNames:
+      - rootcoord

tests/python_client/chaos/chaos_objects/container_kill/chaos_standalone_container_kill.yaml

@@ -0,0 +1,24 @@
+kind: Schedule
+apiVersion: chaos-mesh.org/v1alpha1
+metadata:
+  name: test-standalone-container-kill
+  namespace: chaos-testing
+spec:
+  schedule: '*/5 * * * * *'
+  startingDeadlineSeconds: 60
+  concurrencyPolicy: Forbid
+  historyLimit: 1
+  type: PodChaos
+  podChaos:
+    selector:
+      namespaces:
+        - chaos-testing
+      labelSelectors:
+        app.kubernetes.io/instance: milvus-chaos
+        app.kubernetes.io/name: milvus
+        component: standalone
+    mode: fixed
+    value: "1"
+    action: container-kill
+    containerNames:
+      - standalone

tests/python_client/chaos/chaos_objects/container_kill/chaos_streamingnode_container_kill.yaml

@@ -0,0 +1,24 @@
+kind: Schedule
+apiVersion: chaos-mesh.org/v1alpha1
+metadata:
+  name: test-streamingnode-container-kill
+  namespace: chaos-testing
+spec:
+  schedule: '*/5 * * * * *'
+  startingDeadlineSeconds: 60
+  concurrencyPolicy: Forbid
+  historyLimit: 1
+  type: PodChaos
+  podChaos:
+    selector:
+      namespaces:
+        - chaos-testing
+      labelSelectors:
+        app.kubernetes.io/instance: milvus-chaos
+        app.kubernetes.io/name: milvus
+        component: streamingnode
+    mode: fixed
+    value: "1"
+    action: container-kill
+    containerNames:
+      - streamingnode

tests/python_client/chaos/chaos_objects/container_kill/testcases.yaml

@@ -0,0 +1,171 @@
+# Container Kill Testcases All-in-one
+
+Collections:
+  -
+    testcase:
+      name: test_querynode_container_kill
+      chaos: chaos_querynode_container_kill.yaml
+      expectation:
+        cluster_1_node:
+          search: fail
+          query: fail
+        cluster_n_nodes:
+          search: degrade
+          query: degrade
+  -
+    testcase:
+      name: test_querycoord_container_kill
+      chaos: chaos_querycoord_container_kill.yaml
+      expectation:
+        cluster_1_node:
+          search: fail
+          query: fail
+        cluster_n_nodes:
+          search: degrade
+          query: degrade
+  -
+    testcase:
+      name: test_datanode_container_kill
+      chaos: chaos_datanode_container_kill.yaml
+      expectation:
+        cluster_1_node:
+          insert: succ
+          flush: fail
+        cluster_n_nodes:
+          insert: degrade
+  -
+    testcase:
+      name: test_datacoord_container_kill
+      chaos: chaos_datacoord_container_kill.yaml
+      expectation:
+        cluster_1_node:
+          insert: succ
+          flush: fail
+        cluster_n_nodes:
+          insert: degrade
+  -
+    testcase:
+      name: test_indexnode_container_kill
+      chaos: chaos_indexnode_container_kill.yaml
+      expectation:
+        cluster_1_node:
+          index: fail
+        cluster_n_nodes:
+          index: degrade
+  -
+    testcase:
+      name: test_indexcoord_container_kill
+      chaos: chaos_indexcoord_container_kill.yaml
+      expectation:
+        cluster_1_node:
+          index: fail
+        cluster_n_nodes:
+          insert: degrade
+  -
+    testcase:
+      name: test_proxy_container_kill
+      chaos: chaos_proxy_container_kill.yaml
+      expectation:
+        cluster_1_node:
+          create: fail
+          insert: fail
+          flush: fail
+          index: fail
+          search: fail
+          query: fail
+        cluster_n_nodes:
+          insert: fail
+  -
+    testcase:
+      name: test_rootcoord_container_kill
+      chaos: chaos_rootcoord_container_kill.yaml
+      expectation:
+        cluster_1_node:
+          create: fail
+          insert: fail
+          flush: fail
+          index: fail
+          search: fail
+          query: fail
+        cluster_n_nodes:
+          insert: degrade
+  -
+    testcase:
+      name: test_etcd_container_kill
+      chaos: chaos_etcd_container_kill.yaml
+      expectation:
+        cluster_1_node:
+          create: fail
+          insert: fail
+          flush: fail
+          index: fail
+          search: fail
+          query: fail
+  -
+    testcase:
+      name: test_minio_container_kill
+      chaos: chaos_minio_container_kill.yaml
+      expectation:
+        cluster_1_node:
+          create: fail
+          insert: fail
+          flush: fail
+          index: fail
+          search: fail
+          query: fail
+  -
+    testcase:
+      name: test_pulsar_container_kill
+      chaos: chaos_pulsar_container_kill.yaml
+      expectation:
+        cluster_1_node:
+          create: fail
+          insert: fail
+          flush: fail
+          index: fail
+          search: fail
+          query: fail
+  -
+    testcase:
+      name: test_kafka_container_kill
+      chaos: chaos_kafka_container_kill.yaml
+      expectation:
+        cluster_1_node:
+          create: fail
+          insert: fail
+          flush: fail
+          index: fail
+          search: fail
+          query: fail
+  -
+    testcase:
+      name: test_standalone_container_kill
+      chaos: chaos_standalone_container_kill.yaml
+      expectation:
+        cluster_1_node:
+          create: fail
+          insert: fail
+          flush: fail
+          index: fail
+          search: fail
+          query: fail
+  -
+    testcase:
+      name: test_mixcoord_container_kill
+      chaos: chaos_mixcoord_container_kill.yaml
+      expectation:
+        cluster_1_node:
+          create: fail
+          insert: fail
+          flush: fail
+          index: fail
+          search: fail
+          query: fail
+  -
+    testcase:
+      name: test_streamingnode_container_kill
+      chaos: chaos_streamingnode_container_kill.yaml
+      expectation:
+        cluster_1_node:
+          search: fail
+          query: fail

tests/python_client/chaos/chaos_objects/io_latency/chaos_etcd_io_latency.yaml

@@ -0,0 +1,21 @@
+kind: IOChaos
+apiVersion: chaos-mesh.org/v1alpha1
+metadata:
+  name: test-etcd-io-latency
+  namespace: chaos-testing
+spec:
+  selector:
+    namespaces:
+      - chaos-testing
+    labelSelectors:
+      app.kubernetes.io/name: etcd
+      app.kubernetes.io/instance: milvus-chaos
+  mode: all
+  action: latency
+  delay: 10ms
+  methods:
+    - read
+    - write
+    - flush
+  percent: 100
+  volumePath: /bitnami/etcd

tests/python_client/chaos/chaos_objects/io_latency/chaos_minio_io_latency.yaml

@@ -0,0 +1,21 @@
+kind: IOChaos
+apiVersion: chaos-mesh.org/v1alpha1
+metadata:
+  name: test-minio-io-latency
+  namespace: chaos-testing
+spec:
+  selector:
+    namespaces:
+      - chaos-testing
+    labelSelectors:
+      app: minio
+      release: milvus-chaos
+  mode: all
+  action: latency
+  delay: 10ms
+  methods:
+    - read
+    - write
+    - flush
+  percent: 100
+  volumePath: /export/

tests/python_client/chaos/chaos_objects/io_latency/chaos_pulsar_io_latency.yaml

@@ -0,0 +1,21 @@
+kind: IOChaos
+apiVersion: chaos-mesh.org/v1alpha1
+metadata:
+  name: test-pulsar-io-latency
+  namespace: chaos-testing
+spec:
+  selector:
+    namespaces:
+      - chaos-testing
+    labelSelectors:
+      component: bookkeeper
+      release: milvus-chaos
+  mode: all
+  action: latency
+  delay: 10ms
+  methods:
+    - read
+    - write
+    - flush
+  percent: 100
+  volumePath: /pulsar/data/bookkeeper/ledgers

tests/python_client/chaos/chaos_objects/io_latency/chaos_woodpecker_io_latency.yaml

@@ -0,0 +1,22 @@
+kind: IOChaos
+apiVersion: chaos-mesh.org/v1alpha1
+metadata:
+  name: test-woodpecker-io-latency
+  namespace: chaos-testing
+spec:
+  selector:
+    namespaces:
+      - chaos-testing
+    labelSelectors:
+      app.kubernetes.io/instance: milvus-chaos
+      app.kubernetes.io/name: milvus
+      component: woodpecker
+  mode: all
+  action: latency
+  delay: 10ms
+  methods:
+    - read
+    - write
+    - flush
+  percent: 100
+  volumePath: /milvus/data

tests/python_client/chaos/chaos_objects/io_latency/testcases.yaml

@@ -0,0 +1,51 @@
+Collections:
+  -
+    testcase:
+      name: test_etcd_io_latency
+      chaos: chaos_etcd_io_latency.yaml
+      expectation:
+        cluster_1_node:
+          create: fail
+          insert: fail
+          flush: fail
+          index: fail
+          search: fail
+          query: fail
+
+  -
+    testcase:
+      name: test_minio_io_latency
+      chaos: chaos_minio_io_latency.yaml
+      expectation:
+        cluster_1_node:
+          create: fail
+          insert: fail
+          flush: fail
+          index: fail
+          search: fail
+          query: fail
+
+  -
+    testcase:
+      name: test_pulsar_io_latency
+      chaos: chaos_pulsar_io_latency.yaml
+      expectation:
+        cluster_1_node:
+          create: fail
+          insert: fail
+          flush: fail
+          index: fail
+          search: fail
+          query: fail
+  -
+    testcase:
+      name: test_woodpecker_io_latency
+      chaos: chaos_woodpecker_io_latency.yaml
+      expectation:
+        cluster_1_node:
+          create: fail
+          insert: fail
+          flush: fail
+          index: fail
+          search: fail
+          query: fail

tests/python_client/chaos/chaos_objects/mem_stress/chaos_datanode_mem_stress.yaml

@@ -0,0 +1,21 @@
+kind: StressChaos
+apiVersion: chaos-mesh.org/v1alpha1
+metadata:
+  name: test-datanode-memory-stress
+  namespace: chaos-testing
+spec:
+  selector:
+    namespaces:
+      - chaos-testing
+    labelSelectors:
+      app.kubernetes.io/instance: milvus-chaos
+      component: datanode
+  mode: all
+  stressors:
+    cpu:
+      workers: 4
+      load: 80
+    memory:
+      workers: 4
+      size: 2048Mi
+  duration: 5m

tests/python_client/chaos/chaos_objects/mem_stress/chaos_etcd_mem_stress.yaml

@@ -0,0 +1,21 @@
+kind: StressChaos
+apiVersion: chaos-mesh.org/v1alpha1
+metadata:
+  name: test-etcd-memory-stress
+  namespace: chaos-testing
+spec:
+  selector:
+    namespaces:
+      - chaos-testing
+    labelSelectors:
+      app.kubernetes.io/instance: milvus-chaos
+      app.kubernetes.io/name: etcd
+  mode: all
+  stressors:
+    cpu:
+      workers: 4
+      load: 80
+    memory:
+      workers: 4
+      size: 2048Mi
+  duration: 5m

tests/python_client/chaos/chaos_objects/mem_stress/chaos_indexnode_mem_stress.yaml

@@ -0,0 +1,21 @@
+kind: StressChaos
+apiVersion: chaos-mesh.org/v1alpha1
+metadata:
+  name: test-indexnode-memory-stress
+  namespace: chaos-testing
+spec:
+  selector:
+    namespaces:
+      - chaos-testing
+    labelSelectors:
+      app.kubernetes.io/instance: pulsar-mem-stress-14-04-25
+      component: indexnode
+  mode: one
+  stressors:
+    cpu:
+      workers: 4
+      load: 80
+    memory:
+      workers: 4
+      size: 2048Mi
+  duration: 5m

tests/python_client/chaos/chaos_objects/mem_stress/chaos_minio_mem_stress.yaml

@@ -0,0 +1,21 @@
+kind: StressChaos
+apiVersion: chaos-mesh.org/v1alpha1
+metadata:
+  name: test-datanode-memory-stress
+  namespace: chaos-testing
+spec:
+  selector:
+    namespaces:
+      - chaos-testing
+    labelSelectors:
+      release: milvus-chaos
+      app: minio
+  mode: one
+  stressors:
+    cpu:
+      workers: 4
+      load: 80
+    memory:
+      workers: 4
+      size: 2048Mi
+  duration: 5m

tests/python_client/chaos/chaos_objects/mem_stress/chaos_proxy_mem_stress.yaml

@@ -0,0 +1,21 @@
+kind: StressChaos
+apiVersion: chaos-mesh.org/v1alpha1
+metadata:
+  name: test-datanode-memory-stress
+  namespace: chaos-testing
+spec:
+  selector:
+    namespaces:
+      - chaos-testing
+    labelSelectors:
+      app.kubernetes.io/instance: milvus-chaos
+      component: proxy
+  mode: one
+  stressors:
+    cpu:
+      workers: 4
+      load: 80
+    memory:
+      workers: 4
+      size: 2048Mi
+  duration: 5m

tests/python_client/chaos/chaos_objects/mem_stress/chaos_pulsar_mem_stress.yaml

@@ -0,0 +1,21 @@
+kind: StressChaos
+apiVersion: chaos-mesh.org/v1alpha1
+metadata:
+  name: test-datanode-memory-stress
+  namespace: chaos-testing
+spec:
+  selector:
+    namespaces:
+      - chaos-testing
+    labelSelectors:
+      release: milvus-chaos
+      app: pulsar
+  mode: one
+  stressors:
+    cpu:
+      workers: 4
+      load: 80
+    memory:
+      workers: 4
+      size: 2048Mi
+  duration: 5m

tests/python_client/chaos/chaos_objects/mem_stress/chaos_querynode_mem_stress.yaml

@@ -0,0 +1,21 @@
+kind: StressChaos
+apiVersion: chaos-mesh.org/v1alpha1
+metadata:
+  name: test-querynode-memory-stress
+  namespace: chaos-testing
+spec:
+  selector:
+    namespaces:
+      - chaos-testing
+    labelSelectors:
+      app.kubernetes.io/instance: milvus-chaos
+      component: querynode
+  mode: one
+  stressors:
+    cpu:
+      workers: 4
+      load: 80
+    memory:
+      workers: 4
+      size: 2048Mi
+  duration: 5m

tests/python_client/chaos/chaos_objects/mem_stress/chaos_standalone_mem_stress.yaml

@@ -0,0 +1,21 @@
+kind: StressChaos
+apiVersion: chaos-mesh.org/v1alpha1
+metadata:
+  name: test-datanode-memory-stress
+  namespace: chaos-testing
+spec:
+  selector:
+    namespaces:
+      - chaos-testing
+    labelSelectors:
+      app.kubernetes.io/instance: milvus-chaos
+      component: standalone
+  mode: one
+  stressors:
+    cpu:
+      workers: 4
+      load: 80
+    memory:
+      workers: 4
+      size: 2048Mi
+  duration: 5m

tests/python_client/chaos/chaos_objects/mem_stress/chaos_woodpecker_mem_stress.yaml

@@ -0,0 +1,22 @@
+kind: StressChaos
+apiVersion: chaos-mesh.org/v1alpha1
+metadata:
+  name: test-woodpecker-memory-stress
+  namespace: chaos-testing
+spec:
+  selector:
+    namespaces:
+      - chaos-testing
+    labelSelectors:
+      app.kubernetes.io/instance: milvus-chaos
+      app.kubernetes.io/name: milvus
+      component: woodpecker
+  mode: one
+  stressors:
+    cpu:
+      workers: 4
+      load: 80
+    memory:
+      workers: 4
+      size: 2048Mi
+  duration: 5m

tests/python_client/chaos/chaos_objects/mem_stress/testcases.yaml

@@ -0,0 +1,94 @@
+# Memory Stress Testcases All-in-one
+#   memory stress
+#     standalone
+#       todo
+#     cluster-1-node
+#       11 pods(querynode, datanode, indexnode, pulsar, etcd, minio)
+#     cluster-n-nodes
+#       todo
+
+Collections:
+  -
+    testcase:
+      name: test_querynode_mem_stress
+      chaos: chaos_querynode_mem_stress.yaml
+      expectation:
+          cluster_1_node:
+            search: fail
+            query: fail
+          cluster_n_nodes:
+            search: degrade
+            query: degrade
+  -
+    testcase:
+      name: test_datanode_mem_stress
+      chaos: chaos_datanode_mem_stress.yaml
+      expectation:
+          cluster_1_node:
+            flush: fail
+          cluster_n_nodes:
+            flush: degrade
+  -
+    testcase:
+      name: test_proxy_mem_stress
+      chaos: chaos_proxy_mem_stress.yaml
+      expectation:
+          cluster_1_node:
+            flush: fail
+          cluster_n_nodes:
+            flush: degrade
+  -
+    testcase:
+      name: test_indexnode_mem_stress
+      chaos: chaos_indexnode_mem_stress.yaml
+      expectation:
+          cluster_1_node:
+            flush: fail
+          cluster_n_nodes:
+            flush: degrade
+
+  -
+    testcase:
+      name: test_pulsar_mem_stress
+      chaos: chaos_pulsar_mem_stress.yaml
+      expectation:
+          cluster_1_node:
+            flush: fail
+          cluster_n_nodes:
+            flush: degrade
+  -
+    testcase:
+      name: test_minio_mem_stress
+      chaos: chaos_minio_mem_stress.yaml
+      expectation:
+          cluster_1_node:
+            flush: fail
+          cluster_n_nodes:
+            flush: degrade
+  -
+    testcase:
+      name: test_etcd_mem_stress
+      chaos: chaos_etcd_mem_stress.yaml
+      expectation:
+          cluster_1_node:
+            flush: fail
+          cluster_n_nodes:
+            flush: degrade
+  -
+    testcase:
+      name: test_standalone_mem_stress
+      chaos: chaos_standalone_mem_stress.yaml
+      expectation:
+          cluster_1_node:
+            flush: fail
+          cluster_n_nodes:
+            flush: degrade
+  -
+    testcase:
+      name: test_woodpecker_mem_stress
+      chaos: chaos_woodpecker_mem_stress.yaml
+      expectation:
+          cluster_1_node:
+            flush: fail
+          cluster_n_nodes:
+            flush: degrade

tests/python_client/chaos/chaos_objects/memory_stress/chaos_datanode_memory_stress.yaml

@@ -0,0 +1,18 @@
+kind: StressChaos
+apiVersion: chaos-mesh.org/v1alpha1
+metadata:
+  name: test-datanode-memory-stress
+  namespace: chaos-testing
+spec:
+  selector:
+    namespaces:
+      - chaos-testing
+    labelSelectors:
+      app.kubernetes.io/name: milvus
+      app.kubernetes.io/instance: mic-memory
+      app.kubernetes.io/component: datanode
+  mode: one
+  stressors:
+    memory:
+      workers: 4
+      size: 512Mi

tests/python_client/chaos/chaos_objects/memory_stress/chaos_etcd_memory_stress.yaml

@@ -0,0 +1,18 @@
+kind: StressChaos
+apiVersion: chaos-mesh.org/v1alpha1
+metadata:
+  name: test-etcd-memory-stress
+  namespace: chaos-testing
+spec:
+  selector:
+    namespaces:
+      - chaos-testing
+    labelSelectors:
+      app.kubernetes.io/name: etcd
+      app.kubernetes.io/instance: mic-memory-etcd
+  mode: all
+  stressors:
+    memory:
+      workers: 4
+      size: 1024Mi
+  duration: 5m

tests/python_client/chaos/chaos_objects/memory_stress/chaos_indexnode_memory_stress.yaml

@@ -0,0 +1,19 @@
+kind: StressChaos
+apiVersion: chaos-mesh.org/v1alpha1
+metadata:
+  name: test-indexnode-memory-stress
+  namespace: chaos-testing
+spec:
+  selector:
+    namespaces:
+      - chaos-testing
+    labelSelectors:
+      app.kubernetes.io/name: milvus
+      app.kubernetes.io/instance: mic-memory
+      app.kubernetes.io/component: indexnode
+  mode: one
+  stressors:
+    memory:
+      workers: 4
+      size: 512Mi
+  duration: 2m

tests/python_client/chaos/chaos_objects/memory_stress/chaos_querynode_memory_stress.yaml

@@ -0,0 +1,20 @@
+kind: StressChaos
+apiVersion: chaos-mesh.org/v1alpha1
+metadata:
+  name: test-querynode-memory-stress
+  namespace: chaos-testing
+spec:
+  selector:
+    namespaces:
+      - chaos-testing
+    labelSelectors:
+      app.kubernetes.io/name: milvus
+      app.kubernetes.io/instance: mic-memory
+      app.kubernetes.io/component: querynode
+  mode: all
+  value: "2"
+  stressors:
+    memory:
+      workers: 4
+      size: 5Gi
+  duration: 5m

tests/python_client/chaos/chaos_objects/memory_stress/chaos_replicas_memory_stress_pods.yaml

@@ -0,0 +1,17 @@
+apiVersion: chaos-mesh.org/v1alpha1
+kind: StressChaos
+metadata:
+  name: test-querynode-memory-stress-replica
+  namespace: chaos-testing
+spec:
+  selector:
+    pods:
+      chaos-testing:
+        - mic-replica-milvus-querynode-86c77dd756-rfw8r
+        - mic-replica-milvus-querynode-86c77dd756-wmtdk
+  mode: all
+  stressors:
+    memory:
+      workers: 4
+      size: 85%
+  duration: 3m

tests/python_client/chaos/chaos_objects/memory_stress/testcases.yaml

@@ -0,0 +1,30 @@
+# Memory Stress Testcases All-in-one
+#   memory stress
+#     standalone
+#       todo
+#     cluster-1-node
+#       11 pods(querynode, datanode, indexnode, pulsar, etcd, minio)
+#     cluster-n-nodes
+#       todo
+
+Collections:
+  -
+    testcase:
+      name: test_querynode_memory_stress
+      chaos: chaos_querynode_memory_stress.yaml
+      expectation:
+          cluster_1_node:
+            search: fail
+            query: fail
+          cluster_n_nodes:
+            search: degrade
+            query: degrade
+  -
+    testcase:
+      name: test_datanode_memory_stress
+      chaos: chaos_datanode_memory_stress.yaml
+      expectation:
+          cluster_1_node:
+            flush: fail
+          cluster_n_nodes:
+            flush: degrade

tests/python_client/chaos/chaos_objects/network_latency/chaos_datacoord_network_latency.yaml

@@ -0,0 +1,26 @@
+kind: NetworkChaos
+apiVersion: chaos-mesh.org/v1alpha1
+metadata:
+  name: test-datacoord-network-latency
+  namespace: chaos-testing
+spec:
+  selector:
+    namespaces:
+      - chaos-testing
+    labelSelectors:
+      app.kubernetes.io/instance: milvus-chaos
+      component: datacoord
+  mode: all
+  action: delay
+  delay:
+    latency: 200ms
+    correlation: '100'
+    jitter: 0ms
+  direction: both
+  target:
+    selector:
+      namespaces:
+        - chaos-testing
+      labelSelectors:
+        app.kubernetes.io/instance: milvus-chaos
+    mode: all

tests/python_client/chaos/chaos_objects/network_latency/chaos_datanode_network_latency.yaml

@@ -0,0 +1,26 @@
+kind: NetworkChaos
+apiVersion: chaos-mesh.org/v1alpha1
+metadata:
+  name: test-datanode-network-latency
+  namespace: chaos-testing
+spec:
+  selector:
+    namespaces:
+      - chaos-testing
+    labelSelectors:
+      app.kubernetes.io/instance: milvus-chaos
+      component: datanode
+  mode: all
+  action: delay
+  delay:
+    latency: 200ms
+    correlation: '100'
+    jitter: 0ms
+  direction: both
+  target:
+    selector:
+      namespaces:
+        - chaos-testing
+      labelSelectors:
+        app.kubernetes.io/instance: milvus-chaos
+    mode: all

tests/python_client/chaos/chaos_objects/network_latency/chaos_etcd_network_latency.yaml

@@ -0,0 +1,26 @@
+kind: NetworkChaos
+apiVersion: chaos-mesh.org/v1alpha1
+metadata:
+  name: test-etcd-network-latency
+  namespace: chaos-testing
+spec:
+  selector:
+    namespaces:
+      - chaos-testing
+    labelSelectors:
+      app.kubernetes.io/instance: milvus-chaos
+      app.kubernetes.io/name: etcd
+  mode: all
+  action: delay
+  delay:
+    latency: 200ms
+    correlation: '100'
+    jitter: 0ms
+  direction: both
+  target:
+    selector:
+      namespaces:
+        - chaos-testing
+      labelSelectors:
+        app.kubernetes.io/instance: milvus-chaos
+    mode: all

tests/python_client/chaos/chaos_objects/network_latency/chaos_indexcoord_network_latency.yaml

@@ -0,0 +1,26 @@
+kind: NetworkChaos
+apiVersion: chaos-mesh.org/v1alpha1
+metadata:
+  name: test-indexcoord-network-latency
+  namespace: chaos-testing
+spec:
+  selector:
+    namespaces:
+      - chaos-testing
+    labelSelectors:
+      app.kubernetes.io/instance: milvus-chaos
+      component: indexcoord
+  mode: all
+  action: delay
+  delay:
+    latency: 200ms
+    correlation: '100'
+    jitter: 0ms
+  direction: both
+  target:
+    selector:
+      namespaces:
+        - chaos-testing
+      labelSelectors:
+        app.kubernetes.io/instance: milvus-chaos
+    mode: all

tests/python_client/chaos/chaos_objects/network_latency/chaos_indexnode_network_latency.yaml

@@ -0,0 +1,26 @@
+kind: NetworkChaos
+apiVersion: chaos-mesh.org/v1alpha1
+metadata:
+  name: test-indexnode-network-latency
+  namespace: chaos-testing
+spec:
+  selector:
+    namespaces:
+      - chaos-testing
+    labelSelectors:
+      app.kubernetes.io/instance: milvus-chaos
+      component: indexnode
+  mode: all
+  action: delay
+  delay:
+    latency: 200ms
+    correlation: '100'
+    jitter: 0ms
+  direction: both
+  target:
+    selector:
+      namespaces:
+        - chaos-testing
+      labelSelectors:
+        app.kubernetes.io/instance: milvus-chaos
+    mode: all

tests/python_client/chaos/chaos_objects/network_latency/chaos_minio_network_latency.yaml

@@ -0,0 +1,26 @@
+kind: NetworkChaos
+apiVersion: chaos-mesh.org/v1alpha1
+metadata:
+  name: test-minio-network-latency
+  namespace: chaos-testing
+spec:
+  selector:
+    namespaces:
+      - chaos-testing
+    labelSelectors:
+      release: milvus-chaos
+      app: minio
+  mode: all
+  action: delay
+  delay:
+    latency: 200ms
+    correlation: '100'
+    jitter: 0ms
+  direction: both
+  target:
+    selector:
+      namespaces:
+        - chaos-testing
+      labelSelectors:
+        app.kubernetes.io/instance: milvus-chaos
+    mode: all

tests/python_client/chaos/chaos_objects/network_latency/chaos_proxy_network_latency.yaml

@@ -0,0 +1,26 @@
+kind: NetworkChaos
+apiVersion: chaos-mesh.org/v1alpha1
+metadata:
+  name: test-proxy-network-latency
+  namespace: chaos-testing
+spec:
+  selector:
+    namespaces:
+      - chaos-testing
+    labelSelectors:
+      app.kubernetes.io/instance: milvus-chaos
+      component: proxy
+  mode: all
+  action: delay
+  delay:
+    latency: 200ms
+    correlation: '100'
+    jitter: 0ms
+  direction: both
+  target:
+    selector:
+      namespaces:
+        - chaos-testing
+      labelSelectors:
+        app.kubernetes.io/instance: milvus-chaos
+    mode: all

tests/python_client/chaos/chaos_objects/network_latency/chaos_pulsar_network_latency.yaml

@@ -0,0 +1,26 @@
+kind: NetworkChaos
+apiVersion: chaos-mesh.org/v1alpha1
+metadata:
+  name: test-pulsar-network-latency
+  namespace: chaos-testing
+spec:
+  selector:
+    namespaces:
+      - chaos-testing
+    labelSelectors:
+      release: milvus-chaos
+      app: pulsar
+  mode: all
+  action: delay
+  delay:
+    latency: 200ms
+    correlation: '100'
+    jitter: 0ms
+  direction: both
+  target:
+    selector:
+      namespaces:
+        - chaos-testing
+      labelSelectors:
+        app.kubernetes.io/instance: milvus-chaos
+    mode: all

tests/python_client/chaos/chaos_objects/network_latency/chaos_querycoord_network_latency.yaml

@@ -0,0 +1,26 @@
+kind: NetworkChaos
+apiVersion: chaos-mesh.org/v1alpha1
+metadata:
+  name: test-querycoord-network-latency
+  namespace: chaos-testing
+spec:
+  selector:
+    namespaces:
+      - chaos-testing
+    labelSelectors:
+      app.kubernetes.io/instance: milvus-chaos
+      component: querycoord
+  mode: all
+  action: delay
+  delay:
+    latency: 200ms
+    correlation: '100'
+    jitter: 0ms
+  direction: both
+  target:
+    selector:
+      namespaces:
+        - chaos-testing
+      labelSelectors:
+        app.kubernetes.io/instance: milvus-chaos
+    mode: all

tests/python_client/chaos/chaos_objects/network_latency/chaos_querynode_network_latency.yaml

@@ -0,0 +1,26 @@
+kind: NetworkChaos
+apiVersion: chaos-mesh.org/v1alpha1
+metadata:
+  name: test-querynode-network-latency
+  namespace: chaos-testing
+spec:
+  selector:
+    namespaces:
+      - chaos-testing
+    labelSelectors:
+      app.kubernetes.io/instance: milvus-chaos
+      component: querynode
+  mode: all
+  action: delay
+  delay:
+    latency: 200ms
+    correlation: '100'
+    jitter: 0ms
+  direction: both
+  target:
+    selector:
+      namespaces:
+        - chaos-testing
+      labelSelectors:
+        app.kubernetes.io/instance: milvus-chaos
+    mode: all