chore: import upstream snapshot with attribution

tests/python_client/rolling_upgrade/README.md

@@ -0,0 +1,126 @@
+# Milvus Rolling Upgrade Tests
+
+This directory contains automated tests for validating Milvus rolling upgrade functionality, ensuring that clusters can be upgraded to new versions without service disruption.
+
+## Overview
+
+Rolling upgrade tests verify that Milvus can perform seamless version upgrades while maintaining:
+- Continuous service availability
+- Data integrity
+- Minimal performance impact
+- Client operation compatibility
+
+## Test Files
+
+### Core Test Suites
+
+1. **test_rolling_update_by_default.py**
+   - Tests default rolling upgrade using Kubernetes CRD
+   - Updates all components simultaneously
+   - Validates cluster health and readiness
+
+2. **test_rolling_update_one_by_one.py**
+   - Tests granular component-by-component upgrades
+   - Default order: indexNode → rootCoord → dataCoord/indexCoord → queryCoord → queryNode → dataNode → proxy
+   - Supports pausing/resuming specific components
+
+### Operation Tests
+
+1. **testcases/test_concurrent_request_operation_for_rolling_update.py**
+   - Validates system behavior under concurrent load during upgrades
+   - Tests insert, search, query, and delete operations
+   - Monitors success rates (>98% threshold) and RTO
+
+2. **testcases/test_single_request_operation_for_rolling_update.py**
+   - Tests individual operation types during upgrades
+   - Includes collection creation, flush, and index operations
+   - Tracks per-operation metrics and failures
+
+### Utilities
+
+- **monitor_rolling_update.py**: Standalone pod monitoring utility
+- **conftest.py**: Test configuration and fixtures
+
+## Configuration
+
+### Milvus CRD Files
+
+- **milvus_crd.yaml**: Standard cluster configuration
+- **milvus_mixcoord_crd.yaml**: MixCoord deployment configuration
+
+### Key Parameters
+
+| Parameter | Description | Default |
+|-----------|-------------|---------|
+| `new_image_repo` | Target image repository | - |
+| `new_image_tag` | Target version tag | - |
+| `components_order` | Component update sequence | See test files |
+| `paused_components` | Components to pause during update | [] |
+| `paused_duration` | Pause duration in seconds | 300 |
+| `request_duration` | Operation test duration | 1800 |
+| `is_check` | Enforce success rate assertions | True |
+
+## Running Tests
+
+### Prerequisites
+
+1. Kubernetes cluster with Milvus operator installed
+2. kubectl configured with cluster access
+3. Python environment with required dependencies
+
+### Basic Usage
+
+```bash
+# Run default rolling upgrade test
+pytest test_rolling_update_by_default.py -v
+
+# Run one-by-one upgrade test
+pytest test_rolling_update_one_by_one.py -v
+
+# Run with custom parameters
+pytest test_rolling_update_one_by_one.py \
+  --new_image_repo=milvusdb/milvus \
+  --new_image_tag=v2.4.1 \
+  --paused_components=queryNode \
+  --paused_duration=600
+```
+
+### Monitor Pod Status
+
+```bash
+# Monitor pods for 10 minutes with 5-second intervals
+python monitor_rolling_update.py --duration 600 --interval 5 --release my-release
+```
+
+## Success Criteria
+
+### During Upgrade
+- **Success Rate**: ≥98% for all operations
+- **RTO**: ≤10 seconds per operation type
+- **Pod Status**: All pods eventually reach Ready state
+
+### After Upgrade
+- **Success Rate**: 100% for all operations
+- **Cluster Health**: All components healthy
+- **Data Integrity**: No data loss or corruption
+
+## Test Results
+
+Test results are saved in parquet format:
+- `upgrade_result_<operation>_<timestamp>.parquet`
+- Contains detailed metrics, failed requests, and RTO data
+
+## Architecture
+
+```
+Rolling Upgrade Process:
+┌─────────────┐     ┌─────────────┐     ┌─────────────┐
+│  Old Image  │ --> │  Upgrading  │ --> │  New Image  │
+└─────────────┘     └─────────────┘     └─────────────┘
+       │                    │                    │
+       v                    v                    v
+  [Running Pods]      [Mixed Pods]         [Updated Pods]
+       │                    │                    │
+       └────────────────────┴────────────────────┘
+                    Continuous Operations
+```

tests/python_client/rolling_upgrade/conftest.py

@@ -0,0 +1,89 @@
+import pytest
+
+
+timeout = 60
+dimension = 128
+delete_timeout = 60
+
+
+def pytest_addoption(parser):
+    parser.addoption(
+        "--release_name",
+        type=str,
+        action="store",
+        default="deploy-test",
+        help="release name for deploy test",
+    )
+    parser.addoption(
+        "--new_image_repo",
+        type=str,
+        action="store",
+        default="harbor.milvus.io/dockerhub/milvusdb/milvus",
+        help="image repo",
+    )
+    parser.addoption(
+        "--new_image_tag",
+        type=str,
+        action="store",
+        default="master-20231031-ab6dbf76",
+        help="image tag",
+    )
+    parser.addoption(
+        "--components_order",
+        type=str,
+        action="store",
+        default="['indexNode', 'rootCoord', ['dataCoord', 'indexCoord'], 'queryCoord', 'queryNode', 'dataNode', 'proxy']",
+        help="components update order",
+    )
+    parser.addoption(
+        "--paused_components",
+        type=str,
+        action="store",
+        default="['queryNode']",
+        help="components will be paused during rolling update",
+    )
+    parser.addoption(
+        "--paused_duration",
+        type=int,
+        action="store",
+        default=300,
+        help="paused duration for rolling update in some components",
+    )
+    parser.addoption(
+        "--prepare_data", action="store", type=bool, default=False, help="prepare_data"
+    )
+
+
+@pytest.fixture
+def release_name(request):
+    return request.config.getoption("--release_name")
+
+
+@pytest.fixture
+def new_image_repo(request):
+    return request.config.getoption("--new_image_repo")
+
+
+@pytest.fixture
+def new_image_tag(request):
+    return request.config.getoption("--new_image_tag")
+
+
+@pytest.fixture
+def components_order(request):
+    return request.config.getoption("--components_order")
+
+
+@pytest.fixture
+def paused_components(request):
+    return request.config.getoption("--paused_components")
+
+
+@pytest.fixture
+def paused_duration(request):
+    return request.config.getoption("--paused_duration")
+
+
+@pytest.fixture
+def prepare_data(request):
+    return request.config.getoption("--prepare_data")

tests/python_client/rolling_upgrade/milvus_crd/milvus_crd.yaml

@@ -0,0 +1,189 @@
+# This is a sample to deploy a milvus cluster using pulsar with minimum cost of resources.
+apiVersion: milvus.io/v1beta1
+kind: Milvus
+metadata:
+  name: rolling-update-test
+  namespace: chaos-testing
+  labels:
+    app: milvus
+spec:
+  mode: cluster
+  config:
+    rootCoord:
+      enableActiveStandby: true
+    dataCoord:
+      enableActiveStandby: true
+    queryCoord:
+      enableActiveStandby: true
+    indexCoord:
+      enableActiveStandby: true
+    #    mixCoord:
+    #      enableActiveStandby: true
+    quotaAndLimits:
+      enable: false
+    log:
+      level: debug
+  components:
+    enableRollingUpdate: true
+    imageUpdateMode: rollingUpgrade
+    image: milvusdb/milvus:2.2.0-20231021-1f972292
+    disableMetric: false
+    standalone:
+      resources:
+        requests:
+          cpu: 4
+          memory: 8Gi
+        limits:
+          cpu: 8
+          memory: 16Gi
+      replicas: 1
+    rootCoord:
+      replicas: 1
+    dataCoord:
+      replicas: 1
+    indexCoord:
+      replicas: 1
+    queryCoord:
+      replicas: 1
+    proxy:
+      replicas: 1
+    dataNode:
+      replicas: 3
+    indexNode:
+      replicas: 3
+    queryNode:
+      resources:
+        requests:
+          cpu: 2
+          memory: 8Gi
+        limits:
+          cpu: 4
+          memory: 16Gi
+      replicas: 3
+  dependencies:
+    msgStreamType: kafka
+    etcd:
+      inCluster:
+        deletionPolicy: Retain
+        pvcDeletion: false
+        values:
+          replicaCount: 3
+          metrics:
+            enabled: true
+            podMonitor:
+              enabled: true
+              namespace: chaos-testing
+    kafka:
+      inCluster:
+        deletionPolicy: Retain
+        pvcDeletion: false
+        values:
+          replicaCount: 3
+          defaultReplicationFactor: 2
+          metrics:
+            kafka:
+              enabled: true
+              serviceMonitor:
+                enabled: true
+            jmx:
+              enabled: true
+    pulsar:
+      inCluster:
+        deletionPolicy: Retain
+        pvcDeletion: false
+        values:
+          components:
+            autorecovery: false
+            functions: false
+            toolset: false
+            pulsar_manager: false
+          monitoring:
+            prometheus: false
+            grafana: false
+            node_exporter: false
+            alert_manager: false
+          proxy:
+            replicaCount: 1
+            resources:
+              requests:
+                cpu: 0.01
+                memory: 256Mi
+            configData:
+              PULSAR_MEM: >
+                -Xms256m -Xmx256m
+              PULSAR_GC: >
+                -XX:MaxDirectMemorySize=256m
+          bookkeeper:
+            replicaCount: 2
+            resources:
+              requests:
+                cpu: 0.01
+                memory: 256Mi
+            configData:
+              PULSAR_MEM: >
+                -Xms256m
+                -Xmx256m
+                -XX:MaxDirectMemorySize=256m
+              PULSAR_GC: >
+                -Dio.netty.leakDetectionLevel=disabled
+                -Dio.netty.recycler.linkCapacity=1024
+                -XX:+UseG1GC -XX:MaxGCPauseMillis=10
+                -XX:+ParallelRefProcEnabled
+                -XX:+UnlockExperimentalVMOptions
+                -XX:+DoEscapeAnalysis
+                -XX:ParallelGCThreads=32
+                -XX:ConcGCThreads=32
+                -XX:G1NewSizePercent=50
+                -XX:+DisableExplicitGC
+                -XX:-ResizePLAB
+                -XX:+ExitOnOutOfMemoryError
+                -XX:+PerfDisableSharedMem
+                -XX:+PrintGCDetails
+          zookeeper:
+            replicaCount: 1
+            resources:
+              requests:
+                cpu: 0.01
+                memory: 256Mi
+            configData:
+              PULSAR_MEM: >
+                -Xms256m
+                -Xmx256m
+              PULSAR_GC: >
+                -Dcom.sun.management.jmxremote
+                -Djute.maxbuffer=10485760
+                -XX:+ParallelRefProcEnabled
+                -XX:+UnlockExperimentalVMOptions
+                -XX:+DoEscapeAnalysis -XX:+DisableExplicitGC
+                -XX:+PerfDisableSharedMem
+                -Dzookeeper.forceSync=no
+          broker:
+            replicaCount: 1
+            resources:
+              requests:
+                cpu: 0.01
+                memory: 256Mi
+            configData:
+              PULSAR_MEM: >
+                -Xms256m
+                -Xmx256m
+              PULSAR_GC: >
+                -XX:MaxDirectMemorySize=256m
+                -Dio.netty.leakDetectionLevel=disabled
+                -Dio.netty.recycler.linkCapacity=1024
+                -XX:+ParallelRefProcEnabled
+                -XX:+UnlockExperimentalVMOptions
+                -XX:+DoEscapeAnalysis
+                -XX:ParallelGCThreads=32
+                -XX:ConcGCThreads=32
+                -XX:G1NewSizePercent=50
+                -XX:+DisableExplicitGC
+                -XX:-ResizePLAB
+                -XX:+ExitOnOutOfMemoryError
+    storage:
+      type: Azure
+      inCluster:
+        deletionPolicy: Retain
+        pvcDeletion: false
+        values:
+          mode: distributed

tests/python_client/rolling_upgrade/milvus_crd/milvus_mixcoord_crd.yaml

@@ -0,0 +1,186 @@
+# This is a sample to deploy a milvus cluster using pulsar with minimum cost of resources.
+apiVersion: milvus.io/v1beta1
+kind: Milvus
+metadata:
+  name: rolling-update-test
+  namespace: chaos-testing
+  labels:
+    app: milvus
+spec:
+  mode: cluster
+  config:
+    rootCoord:
+      enableActiveStandby: true
+    dataCoord:
+      enableActiveStandby: true
+    queryCoord:
+      enableActiveStandby: true
+    indexCoord:
+      enableActiveStandby: true
+    mixCoord:
+      enableActiveStandby: true
+    quotaAndLimits:
+      enable: false
+    log:
+      level: debug
+  components:
+    enableRollingUpdate: true
+    imageUpdateMode: rollingUpgrade
+    image: milvusdb/milvus:2.2.0-20231021-1f972292
+    disableMetric: false
+    mixCoord:
+      replicas: 1
+    proxy:
+      replicas: 1
+    dataNode:
+      resources:
+        requests:
+          cpu: 4
+          memory: 64Gi
+        limits:
+          cpu: 8
+          memory: 64Gi
+      replicas: 1
+      podAnnotations:
+        pyroscope.io/application-name: "rolling-update-test-datanode"
+        pyroscope.io/port: "9091"
+        pyroscope.io/scrape: "true"
+    indexNode:
+      replicas: 3
+    queryNode:
+      replicas: 3
+  dependencies:
+    msgStreamType: kafka
+    etcd:
+      inCluster:
+        deletionPolicy: Retain
+        pvcDeletion: false
+        values:
+          replicaCount: 3
+          metrics:
+            enabled: true
+            podMonitor:
+              enabled: true
+              namespace: chaos-testing
+    kafka:
+      inCluster:
+        deletionPolicy: Retain
+        pvcDeletion: false
+        values:
+          replicaCount: 3
+          defaultReplicationFactor: 2
+          metrics:
+            kafka:
+              enabled: true
+              serviceMonitor:
+                enabled: true
+            jmx:
+              enabled: true
+    pulsar:
+      inCluster:
+        deletionPolicy: Retain
+        pvcDeletion: false
+        values:
+          components:
+            autorecovery: false
+            functions: false
+            toolset: false
+            pulsar_manager: false
+          monitoring:
+            prometheus: false
+            grafana: false
+            node_exporter: false
+            alert_manager: false
+          proxy:
+            podMonitor:
+              enabled: true
+            replicaCount: 1
+            resources:
+              requests:
+                cpu: 0.01
+                memory: 256Mi
+            configData:
+              PULSAR_MEM: >
+                -Xms256m -Xmx256m
+              PULSAR_GC: >
+                -XX:MaxDirectMemorySize=256m
+          bookkeeper:
+            podMonitor:
+              enabled: true
+            replicaCount: 2
+            resources:
+              requests:
+                cpu: 0.01
+                memory: 256Mi
+            configData:
+              PULSAR_MEM: >
+                -Xms256m
+                -Xmx256m
+                -XX:MaxDirectMemorySize=256m
+              PULSAR_GC: >
+                -Dio.netty.leakDetectionLevel=disabled
+                -Dio.netty.recycler.linkCapacity=1024
+                -XX:+UseG1GC -XX:MaxGCPauseMillis=10
+                -XX:+ParallelRefProcEnabled
+                -XX:+UnlockExperimentalVMOptions
+                -XX:+DoEscapeAnalysis
+                -XX:ParallelGCThreads=32
+                -XX:ConcGCThreads=32
+                -XX:G1NewSizePercent=50
+                -XX:+DisableExplicitGC
+                -XX:-ResizePLAB
+                -XX:+ExitOnOutOfMemoryError
+                -XX:+PerfDisableSharedMem
+                -XX:+PrintGCDetails
+          zookeeper:
+            podMonitor:
+              enabled: true
+            replicaCount: 1
+            resources:
+              requests:
+                cpu: 0.01
+                memory: 256Mi
+            configData:
+              PULSAR_MEM: >
+                -Xms256m
+                -Xmx256m
+              PULSAR_GC: >
+                -Dcom.sun.management.jmxremote
+                -Djute.maxbuffer=10485760
+                -XX:+ParallelRefProcEnabled
+                -XX:+UnlockExperimentalVMOptions
+                -XX:+DoEscapeAnalysis -XX:+DisableExplicitGC
+                -XX:+PerfDisableSharedMem
+                -Dzookeeper.forceSync=no
+          broker:
+            podMonitor:
+              enabled: true
+            replicaCount: 1
+            resources:
+              requests:
+                cpu: 0.01
+                memory: 256Mi
+            configData:
+              PULSAR_MEM: >
+                -Xms256m
+                -Xmx256m
+              PULSAR_GC: >
+                -XX:MaxDirectMemorySize=256m
+                -Dio.netty.leakDetectionLevel=disabled
+                -Dio.netty.recycler.linkCapacity=1024
+                -XX:+ParallelRefProcEnabled
+                -XX:+UnlockExperimentalVMOptions
+                -XX:+DoEscapeAnalysis
+                -XX:ParallelGCThreads=32
+                -XX:ConcGCThreads=32
+                -XX:G1NewSizePercent=50
+                -XX:+DisableExplicitGC
+                -XX:-ResizePLAB
+                -XX:+ExitOnOutOfMemoryError
+    storage:
+      type: Azure
+      inCluster:
+        deletionPolicy: Retain
+        pvcDeletion: false
+        values:
+          mode: distributed

tests/python_client/rolling_upgrade/monitor_rolling_update.py

@@ -0,0 +1,47 @@
+import argparse
+import subprocess
+import time
+from loguru import logger as log
+
+
+def run_kubectl_get_pod(duration, interval, release_name):
+    end_time = time.time() + duration
+    while time.time() < end_time:
+        cmd = f"kubectl get pod |grep {release_name}"
+        res = subprocess.Popen(
+            cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE
+        )
+        stdout, stderr = res.communicate()
+        output = stdout.decode("utf-8")
+        log.info(f"{cmd}\n{output}\n")
+        time.sleep(interval)
+
+
+if __name__ == "__main__":
+    parser = argparse.ArgumentParser(
+        description='Script to run "kubectl get pod" command at regular intervals'
+    )
+    parser.add_argument(
+        "-d",
+        "--duration",
+        type=int,
+        default=600,
+        help="Duration in seconds (default: 600)",
+    )
+    parser.add_argument(
+        "-i",
+        "--interval",
+        type=int,
+        default=5,
+        help="Interval in seconds (default: 30)",
+    )
+    parser.add_argument(
+        "-n",
+        "--release_name",
+        type=str,
+        default="",
+        help='release name (default: "None")',
+    )
+    args = parser.parse_args()
+
+    run_kubectl_get_pod(args.duration, args.interval, args.release_name)

tests/python_client/rolling_upgrade/test_rolling_update_by_default.py

@@ -0,0 +1,89 @@
+from pprint import pformat
+from pathlib import Path
+import subprocess
+import pytest
+from time import sleep
+import yaml
+from utils.util_log import test_log as log
+from common.common_type import CaseLabel
+from chaos import constants
+
+
+class TestBase:
+    expect_create = constants.SUCC
+    expect_insert = constants.SUCC
+    expect_flush = constants.SUCC
+    expect_index = constants.SUCC
+    expect_search = constants.SUCC
+    expect_query = constants.SUCC
+    host = "127.0.0.1"
+    port = 19530
+    _chaos_config = None
+    health_checkers = {}
+
+
+def run_cmd(cmd):
+    log.info(f"cmd: {cmd}")
+    res = subprocess.Popen(
+        cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE
+    )
+    stdout, stderr = res.communicate()
+    output = stdout.decode("utf-8")
+    log.info(f"{cmd}\n{output}\n")
+    return output
+
+
+class TestOperations(TestBase):
+    @pytest.mark.tags(CaseLabel.L3)
+    def test_operations(self, new_image_repo, new_image_tag):
+        log.info("*********************Rolling Update Start**********************")
+        origin_file_path = f"{str(Path(__file__).parent)}/milvus_crd.yaml"
+        with open(origin_file_path, "r") as f:
+            config = yaml.load(f, Loader=yaml.FullLoader)
+        target_image = f"{new_image_repo}:{new_image_tag}"
+        config["spec"]["components"]["image"] = target_image
+        config["spec"]["components"]["imageUpdateMode"] = "rollingUpgrade"
+        log.info(f"config: {pformat(config['spec']['components'])}")
+        components = [
+            "indexNode",
+            "rootCoord",
+            "dataCoord",
+            "indexCoord",
+            "queryCoord",
+            "dataNode",
+            "queryNode",
+            "proxy",
+            "standalone",
+            "mixCoord",
+        ]
+        # delete image in specific component
+        for c in components:
+            if c in config["spec"]["components"]:
+                config["spec"]["components"][c]["image"] = ""
+        # save config to a modified file
+        modified_file_path = f"{str(Path(__file__).parent)}/milvus_crd_modified.yaml"
+        with open(modified_file_path, "w") as f:
+            yaml.dump(config, f, default_flow_style=False, sort_keys=False)
+        kind = config["kind"]
+        meta_name = config["metadata"]["name"]
+        cmd = f"kubectl patch {kind} {meta_name} --patch-file {modified_file_path} --type merge"
+        run_cmd(cmd)
+        # check pod status
+        log.info("wait 10s after rolling update patch")
+        sleep(10)
+        cmd = f"kubectl get pod|grep {meta_name}"
+        run_cmd(cmd)
+        # check milvus status
+        ready = False
+        while ready is False:
+            cmd = f"kubectl get pod|grep {meta_name}"
+            run_cmd(cmd)
+            sleep(10)
+            cmd = f"kubectl get mi {meta_name}"
+            output = run_cmd(cmd)
+            if "True" in output:
+                ready = True
+            else:
+                log.info("wait 10s for milvus ready and updated")
+                sleep(10)
+        log.info("*********************Test Completed**********************")

tests/python_client/rolling_upgrade/test_rolling_update_one_by_one.py

@@ -0,0 +1,283 @@
+from pprint import pformat
+from pathlib import Path
+import subprocess
+import pytest
+from time import sleep
+import yaml
+from datetime import datetime
+from utils.util_log import test_log as log
+from common.common_type import CaseLabel
+from chaos import constants
+from common.cus_resource_opts import CustomResourceOperations as CusResource
+import time
+from kubernetes import client, config
+
+
+class TestBase:
+    expect_create = constants.SUCC
+    expect_insert = constants.SUCC
+    expect_flush = constants.SUCC
+    expect_index = constants.SUCC
+    expect_search = constants.SUCC
+    expect_query = constants.SUCC
+    host = "127.0.0.1"
+    port = 19530
+    _chaos_config = None
+    health_checkers = {}
+
+
+def interrupt_rolling(release_name, component, timeout=60):
+    # get the querynode pod name which age is the newest
+    cmd = f"kubectl get pod -n chaos-testing|grep {release_name}|grep {component}|awk '{{print $1}}'"
+    _ = run_cmd(cmd)
+
+    chaos_config = {}
+    # apply chaos object
+    chaos_res = CusResource(
+        kind=chaos_config["kind"],
+        group=constants.CHAOS_GROUP,
+        version=constants.CHAOS_VERSION,
+        namespace=constants.CHAOS_NAMESPACE,
+    )
+    chaos_res.create(chaos_config)
+    create_time = datetime.fromtimestamp(time.time()).strftime("%Y-%m-%d %H:%M:%S.%f")
+    log.info(f"chaos injected at {create_time}")
+
+
+def pause_and_resume_rolling(
+    metadata_name, deployment_name, namespace, updated_pod_count, pause_seconds
+):
+    """
+    Method: pause and resume the rolling update of a Deployment
+    Params:
+        metadata_name: name of the Milvus custom resource
+        deployment_name: name of the Deployment to pause and resume (choices: querynode, indexnode, datanode)
+        namespace: namespace that the Milvus custom resource is running in
+        updated_pod_count: number of Pods to update before pausing the rolling update
+        pause_seconds: number of seconds to pause the rolling update
+    example:
+        pause_and_resume_rolling("milvus-cluster", "milvus-cluster-querynode", "default", 1, 60)
+    """
+    # Load Kubernetes configuration
+    config.load_kube_config()
+    api_instance = client.AppsV1Api()
+
+    # resuem the rolling update of a Deployment if it is paused
+    config = {"spec": {"components": {"paused": False}}}
+    with open("milvus_resume.yaml", "w") as f:
+        yaml.dump(config, f, default_flow_style=False, sort_keys=False)
+    cmd = f"kubectl patch milvus {metadata_name} --patch-file milvus_resume.yaml --type merge"
+    run_cmd(cmd)
+    log.info(f"Resumed milvus {metadata_name} deployment {deployment_name}")
+
+    # Monitor the number of updated Pods
+    while True:
+        res = api_instance.read_namespaced_deployment(deployment_name, namespace)
+        log.info(f"deployment {deployment_name} status: {res.status}")
+
+        updated_replicas = api_instance.read_namespaced_deployment(
+            deployment_name, namespace
+        ).status.updated_replicas
+
+        replicas = api_instance.read_namespaced_deployment(
+            deployment_name, namespace
+        ).status.replicas
+        log.info(f"updated_replicas: {updated_replicas}, replicas: {replicas}")
+        if updated_replicas is None:
+            updated_replicas = 0
+            log.debug(
+                f"updated_replicas: {updated_replicas}, replicas: {replicas}, no replicas updated, keep waiting"
+            )
+        if updated_replicas >= updated_pod_count and updated_replicas < replicas:
+            log.debug(
+                f"updated_replicas: {updated_replicas}, replicas: {replicas}, sastisfy the condition to pause the rolling update"
+            )
+            break
+        if updated_replicas >= replicas:
+            log.debug(
+                f"updated_replicas: {updated_replicas}, replicas: {replicas}, rolling update completed, no need to pause"
+            )
+            return
+        time.sleep(1)
+    # Pause the Deployment's rolling update by patching the custom resource
+    config = {"spec": {"components": {"paused": True}}}
+    with open("milvus_pause.yaml", "w") as f:
+        yaml.dump(config, f, default_flow_style=False, sort_keys=False)
+    time.sleep(1)
+    cmd = f"kubectl patch milvus {metadata_name} --patch-file milvus_pause.yaml --type merge"
+    run_cmd(cmd)
+    log.info(
+        f"Paused Milvus {metadata_name} deployment {deployment_name} after updating {updated_pod_count} replicas. "
+        f"Waiting for {pause_seconds} seconds..."
+    )
+    # Wait for the specified pause time
+    t0 = time.time()
+    cnt = 0
+    while time.time() - t0 < pause_seconds:
+        cmd = f"kubectl get milvus {metadata_name} -o json|jq .spec.components"
+        run_cmd(cmd)
+        res = api_instance.read_namespaced_deployment(deployment_name, namespace)
+        log.debug(f"deployment {deployment_name} status: {res.status}")
+        time.sleep(10)
+        cnt += 1
+        # show progress every 60s
+        if cnt % 6 == 0:
+            log.info(f"progress: paused for {cnt * 10}s / {pause_seconds}s")
+    # Resume the Deployment's rolling update
+    config = {"spec": {"components": {"paused": False}}}
+    with open("milvus_resume.yaml", "w") as f:
+        yaml.dump(config, f, default_flow_style=False, sort_keys=False)
+    cmd = f"kubectl patch milvus {metadata_name} --patch-file milvus_resume.yaml --type merge"
+    run_cmd(cmd)
+    log.info(f"Resumed milvus {metadata_name} deployment {deployment_name}")
+
+
+def run_cmd(cmd):
+    log.info(f"cmd: {cmd}")
+    res = subprocess.Popen(
+        cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE
+    )
+    stdout, stderr = res.communicate()
+    output = stdout.decode("utf-8")
+    log.info(f"{cmd}\n{output}\n")
+    return output
+
+
+def check_querynode_upgrade_complete(release_name, namespace):
+    cmd = (
+        f"kubectl get deployment -n {namespace} | grep {release_name} | grep querynode"
+    )
+    output = run_cmd(cmd)
+    log.info(f"querynode deployment status: {output}")
+    deployments = [line.split() for line in output.strip().split("\n")]
+    status = []
+    current_deployment = ""
+    name_list = []
+    for deployment in deployments:
+        name, ready, *_ = deployment
+        name_list.append(name)
+    name_list = sorted(name_list)
+    current_deployment = name_list[-1]
+
+    for deployment in deployments:
+        name, ready, *_ = deployment
+        ready_pods, total_pods = map(int, ready.split("/"))
+        # the old querynode deployment should have 0/0 pods ready
+        if name != current_deployment:
+            if ready_pods == 0:
+                status.append(True)
+            else:
+                status.append(False)
+        else:
+            if ready_pods == total_pods:
+                status.append(True)
+            else:
+                status.append(False)
+    return all(status)
+
+
+class TestOperations(TestBase):
+    @pytest.mark.tags(CaseLabel.L3)
+    def test_operations(
+        self,
+        new_image_repo,
+        new_image_tag,
+        components_order,
+        paused_components,
+        paused_duration,
+    ):
+        log.info("*********************Rolling Update Start**********************")
+        paused_components = eval(paused_components)
+        log.info(f"paused_components: {paused_components}")
+        paused_duration = int(paused_duration)
+        origin_file_path = f"{str(Path(__file__).parent)}/milvus_crd.yaml"
+        log.info(f"origin_file_path: {origin_file_path}")
+        with open(origin_file_path, "r") as f:
+            config = yaml.load(f, Loader=yaml.FullLoader)
+        log.info(f"config: {pformat(config['spec']['components'])}")
+        target_image = f"{new_image_repo}:{new_image_tag}"
+        if "image" in config["spec"]["components"]:
+            del config["spec"]["components"]["image"]
+        config["spec"]["components"]["imageUpdateMode"] = "all"
+        log.info(f"config: {pformat(config['spec']['components'])}")
+        # save config to a modified file
+        modified_file_path = f"{str(Path(__file__).parent)}/milvus_crd_modified.yaml"
+        with open(modified_file_path, "w") as f:
+            yaml.dump(config, f, default_flow_style=False, sort_keys=False)
+        kind = config["kind"]
+        meta_name = config["metadata"]["name"]
+        namespace = config["metadata"]["namespace"]
+        # default is ['indexNode', 'rootCoord', ['dataCoord', 'indexCoord'], 'queryCoord', 'queryNode', 'dataNode', 'proxy']
+        components = eval(components_order)
+        log.info(f"update order: {components}")
+        component_time_map = {}
+        for component in components:
+            prefix = f"[update image for {component}]"
+            # load config and update
+            with open(modified_file_path, "r") as f:
+                config = yaml.load(f, Loader=yaml.FullLoader)
+            if isinstance(component, list):
+                for c in component:
+                    config["spec"]["components"][c]["image"] = target_image
+            else:
+                config["spec"]["components"][component]["image"] = target_image
+            log.info(prefix + f"config: {pformat(config['spec']['components'])}")
+            # save config to file
+            with open(modified_file_path, "w") as f:
+                yaml.dump(config, f, default_flow_style=False, sort_keys=False)
+            log.info(f"update image for component {component}")
+            cmd = f"kubectl patch {kind} {meta_name} --patch-file {modified_file_path} --type merge"
+            run_cmd(cmd)
+            component_time_map[str(component)] = datetime.now()
+
+            # if component in paused_components:
+            #     log.info(f"start to interrupt rolling update for {component}")
+            #     deploy_name = f"{meta_name}-milvus-{component.lower()}" if "milvus" not in component else f"{meta_name}-{component.lower()}"
+            #     cmd = f"kubectl get deployment | grep {deploy_name}"
+            #     run_cmd(cmd)
+            #     pause_and_resume_rolling(meta_name, deploy_name, namespace, 2, paused_duration)
+            # check pod status
+            log.info(prefix + "wait 10s after rolling update patch")
+            sleep(10)
+            cmd = f"kubectl get pod|grep {meta_name}"
+            run_cmd(cmd)
+            # check milvus status
+            ready = False
+            while ready is False:
+                cmd = f"kubectl get pod|grep {meta_name}"
+                run_cmd(cmd)
+                sleep(10)
+                cmd = f"kubectl get mi |grep {meta_name}"
+                output = run_cmd(cmd)
+                log.info(f"output: {output}")
+
+                if "True" in output and "Healthy" in output:
+                    # Check if the status remains stable for 1 minute
+                    stable = True
+                    for _ in range(
+                        6
+                    ):  # Check every 10 seconds, 6 times in total (60 seconds)
+                        sleep(10)
+                        output = run_cmd(cmd)
+                        log.info(f"Re-check output: {output}")
+                        if "True" not in output or "Healthy" not in output:
+                            stable = False
+                            break
+                    if component == "queryNode":
+                        log.info(prefix + "add additional check for queryNode")
+                        while not check_querynode_upgrade_complete(
+                            meta_name, namespace
+                        ):
+                            print("Querynode upgrade not complete, waiting...")
+                            time.sleep(10)
+
+                    if stable:
+                        ready = True
+                    else:
+                        log.info(prefix + "status not stable, continue waiting")
+                else:
+                    log.info(prefix + "wait 10s for milvus ready")
+                    sleep(10)
+            sleep(60)
+        log.info(f"rolling update time: {component_time_map}")
+        log.info("*********************Test Completed**********************")

tests/python_client/rolling_upgrade/testcases/test_concurrent_request_operation_for_rolling_update.py

@@ -0,0 +1,160 @@
+from pathlib import Path
+import pytest
+from time import sleep
+from yaml import full_load
+from pymilvus import connections
+from chaos.checker import InsertChecker, SearchChecker, QueryChecker, DeleteChecker, Op
+from utils.util_k8s import wait_pods_ready
+from utils.util_log import test_log as log
+from chaos import chaos_commons as cc
+from common.common_type import CaseLabel
+from common import common_func as cf
+from chaos.chaos_commons import assert_statistic
+from chaos import constants
+import pandas as pd
+
+
+class TestBase:
+    expect_create = constants.SUCC
+    expect_insert = constants.SUCC
+    expect_flush = constants.SUCC
+    expect_index = constants.SUCC
+    expect_search = constants.SUCC
+    expect_query = constants.SUCC
+    host = "127.0.0.1"
+    port = 19530
+    _chaos_config = None
+    health_checkers = {}
+
+
+class TestOperations(TestBase):
+    @pytest.fixture(scope="function", autouse=True)
+    def connection(self, host, port, user, password, minio_host):
+        if user and password:
+            # log.info(f"connect to {host}:{port} with user {user} and password {password}")
+            connections.connect(
+                "default",
+                host=host,
+                port=port,
+                user=user,
+                password=password,
+            )
+        else:
+            connections.connect("default", host=host, port=port)
+        if connections.has_connection("default") is False:
+            raise Exception("no connections")
+        log.info("connect to milvus successfully")
+        self.host = host
+        self.port = port
+        self.user = user
+        self.password = password
+        self.minio_endpoint = f"{minio_host}:9000"
+
+    def init_health_checkers(self, collection_name=None):
+        c_name = collection_name
+        schema = cf.gen_default_collection_schema(auto_id=False)
+
+        checkers = {
+            Op.insert: InsertChecker(collection_name=c_name, schema=schema),
+            Op.search: SearchChecker(collection_name=c_name, schema=schema),
+            Op.query: QueryChecker(collection_name=c_name, schema=schema),
+            Op.delete: DeleteChecker(collection_name=c_name, schema=schema),
+            # Op.bulk_insert: BulkInsertChecker(collection_name=c_name, schema=schema)
+        }
+        self.health_checkers = checkers
+        for k, v in self.health_checkers.items():
+            if k in [Op.bulk_insert]:
+                files = v.prepare_bulk_insert_data(
+                    nb=3000, minio_endpoint=self.minio_endpoint
+                )
+                v.update(files=files)
+
+    @pytest.mark.tags(CaseLabel.L3)
+    def test_operations(self, request_duration, is_check, prepare_data):
+        # start the monitor threads to check the milvus ops
+        log.info("*********************Test Start**********************")
+        log.info(connections.get_connection_addr("default"))
+        c_name = cf.gen_unique_str("Checker_")
+        self.init_health_checkers(collection_name=c_name)
+
+        log.info("*********************Load Start**********************")
+        cc.start_monitor_threads(self.health_checkers)
+
+        # wait request_duration
+        request_duration = (
+            request_duration.replace("h", "*3600+")
+            .replace("m", "*60+")
+            .replace("s", "")
+        )
+        if request_duration[-1] == "+":
+            request_duration = request_duration[:-1]
+        request_duration = eval(request_duration)
+        for i in range(10):
+            sleep(request_duration // 10)
+            for k, v in self.health_checkers.items():
+                v.check_result()
+        for k, v in self.health_checkers.items():
+            v.pause()
+        for k, v in self.health_checkers.items():
+            v.check_result()
+        for k, v in self.health_checkers.items():
+            log.info(f"{k} failed request: {v.fail_records}")
+        for k, v in self.health_checkers.items():
+            log.info(f"{k} rto: {v.get_rto()}")
+
+        # save result to parquet use pandas
+        result = []
+        for k, v in self.health_checkers.items():
+            data = {
+                "op": str(k),
+                "failed request ts": [x[2] for x in v.fail_records],
+                "failed request order": [x[1] for x in v.fail_records],
+                "rto": v.get_rto(),
+            }
+            result.append(data)
+        df = pd.DataFrame(result)
+        log.info(f"result: {df}")
+        # save result to parquet
+        file_name = "/tmp/ci_logs/concurrent_request_result.parquet"
+        Path(file_name).parent.mkdir(parents=True, exist_ok=True)
+        df.to_parquet(file_name)
+        if is_check:
+            assert_statistic(self.health_checkers, succ_rate_threshold=0.98)
+            # get each checker's rto
+            for k, v in self.health_checkers.items():
+                log.info(f"{k} rto: {v.get_rto()}")
+                rto = v.get_rto()
+                rto_threshold = 10
+                pytest.assume(
+                    rto <= rto_threshold,
+                    f"{k} rto expect {rto_threshold}s but get {rto}s",
+                )
+
+            # if Op.insert in self.health_checkers:
+            #     # verify the no insert data loss
+            #     log.info("*********************Verify Data Completeness**********************")
+            #     self.health_checkers[Op.insert].verify_data_completeness()
+
+        #
+        for k, v in self.health_checkers.items():
+            v.reset()
+
+        # wait all pod running
+        file_path = f"{str(Path(__file__).parent.parent.parent)}/deploy/milvus_crd.yaml"
+        with open(file_path, "r") as f:
+            config = full_load(f)
+        meta_name = config["metadata"]["name"]
+        label_selector = f"app.kubernetes.io/instance={meta_name}"
+        is_ready = wait_pods_ready("chaos-testing", label_selector)
+        pytest.assume(is_ready is True, f"expect all pods ready but got {is_ready}")
+        cc.start_monitor_threads(self.health_checkers)
+        sleep(120)
+        log.info("check succ rate after rolling update finished")
+        for k, v in self.health_checkers.items():
+            v.check_result()
+        for k, v in self.health_checkers.items():
+            log.info(f"{k} failed request: {v.fail_records}")
+        for k, v in self.health_checkers.items():
+            log.info(f"{k} rto: {v.get_rto()}")
+        assert_statistic(self.health_checkers, succ_rate_threshold=1.0)
+        log.info("*********************Test Completed**********************")

tests/python_client/rolling_upgrade/testcases/test_single_request_operation_for_rolling_update.py

@@ -0,0 +1,197 @@
+from pathlib import Path
+import pytest
+from time import sleep
+from yaml import full_load
+from pymilvus import connections
+from chaos.checker import (
+    CollectionCreateChecker,
+    InsertChecker,
+    FlushChecker,
+    SearchChecker,
+    QueryChecker,
+    IndexCreateChecker,
+    DeleteChecker,
+    Op,
+)
+from utils.util_k8s import wait_pods_ready
+from utils.util_log import test_log as log
+from chaos import chaos_commons as cc
+from common.common_type import CaseLabel
+from common import common_func as cf
+from chaos.chaos_commons import assert_statistic
+from chaos import constants
+import pandas as pd
+
+
+class TestBase:
+    expect_create = constants.SUCC
+    expect_insert = constants.SUCC
+    expect_flush = constants.SUCC
+    expect_index = constants.SUCC
+    expect_search = constants.SUCC
+    expect_query = constants.SUCC
+    host = "127.0.0.1"
+    port = 19530
+    _chaos_config = None
+    health_checkers = {}
+
+
+class TestOperations(TestBase):
+    @pytest.fixture(scope="function", autouse=True)
+    def connection(self, host, port, user, password, minio_host):
+        if user and password:
+            # log.info(f"connect to {host}:{port} with user {user} and password {password}")
+            connections.connect(
+                "default",
+                host=host,
+                port=port,
+                user=user,
+                password=password,
+            )
+        else:
+            connections.connect("default", host=host, port=port)
+        if connections.has_connection("default") is False:
+            raise Exception("no connections")
+        log.info("connect to milvus successfully")
+        self.host = host
+        self.port = port
+        self.user = user
+        self.password = password
+        self.minio_endpoint = f"{minio_host}:9000"
+
+    def init_health_checkers(self, collection_name=None):
+        c_name = collection_name
+        schema = cf.gen_default_collection_schema(auto_id=False)
+
+        checkers = {
+            Op.create: CollectionCreateChecker(collection_name=None, schema=schema),
+            Op.insert: InsertChecker(collection_name=c_name, schema=schema),
+            Op.flush: FlushChecker(collection_name=c_name, schema=schema),
+            Op.index: IndexCreateChecker(collection_name=None, schema=schema),
+            Op.search: SearchChecker(collection_name=c_name, schema=schema),
+            Op.query: QueryChecker(collection_name=c_name, schema=schema),
+            Op.delete: DeleteChecker(collection_name=c_name, schema=schema),
+            # Op.bulk_insert: BulkInsertChecker(collection_name=c_name, schema=schema)
+        }
+        self.health_checkers = checkers
+        for k, v in self.health_checkers.items():
+            if k in [Op.bulk_insert]:
+                files = v.prepare_bulk_insert_data(
+                    nb=3000, minio_endpoint=self.minio_endpoint
+                )
+                v.update(files=files)
+
+    @pytest.mark.tags(CaseLabel.L3)
+    def test_operations(self, request_duration, is_check, prepare_data):
+        # start the monitor threads to check the milvus ops
+        log.info("*********************Test Start**********************")
+        log.info(connections.get_connection_addr("default"))
+        c_name = None
+        self.init_health_checkers(collection_name=c_name)
+        # # prepare data by bulk insert
+        # if prepare_data:
+        #     log.info("*********************Prepare Data by bulk insert**********************")
+        #     for k, v in self.health_checkers.items():
+        #         if k in [Op.search, Op.query]:
+        #             log.info(f"prepare bulk insert data for {k}")
+        #             v.prepare_bulk_insert_data(minio_endpoint=self.minio_endpoint)
+        #             completed = False
+        #             retry_times = 0
+        #             while not completed and retry_times < 3:
+        #                 completed, result = v.do_bulk_insert()
+        #                 if not completed:
+        #                     log.info(f"do bulk insert failed: {result}")
+        #                     retry_times += 1
+        #                     sleep(5)
+        #             # wait for index building complete
+        #             utility.wait_for_index_building_complete(v.c_name, timeout=120)
+        #             res = utility.index_building_progress(v.c_name)
+        #             index_completed = res["pending_index_rows"] == 0
+        #             while not index_completed:
+        #                 time.sleep(10)
+        #                 res = utility.index_building_progress(v.c_name)
+        #                 log.info(f"index building progress: {res}")
+        #                 index_completed = res["pending_index_rows"] == 0
+        #             log.info(f"index building progress: {res}")
+
+        log.info("*********************Load Start**********************")
+        cc.start_monitor_threads(self.health_checkers)
+
+        # wait request_duration
+        request_duration = (
+            request_duration.replace("h", "*3600+")
+            .replace("m", "*60+")
+            .replace("s", "")
+        )
+        if request_duration[-1] == "+":
+            request_duration = request_duration[:-1]
+        request_duration = eval(request_duration)
+        for i in range(10):
+            sleep(request_duration // 10)
+            for k, v in self.health_checkers.items():
+                v.check_result()
+        for k, v in self.health_checkers.items():
+            v.pause()
+        for k, v in self.health_checkers.items():
+            v.check_result()
+        for k, v in self.health_checkers.items():
+            log.info(f"{k} failed request: {v.fail_records}")
+        for k, v in self.health_checkers.items():
+            log.info(f"{k} rto: {v.get_rto()}")
+
+            # save result to parquet use pandas
+        result = []
+        for k, v in self.health_checkers.items():
+            data = {
+                "op": str(k),
+                "failed request ts": [x[2] for x in v.fail_records],
+                "failed request order": [x[1] for x in v.fail_records],
+                "rto": v.get_rto(),
+            }
+            result.append(data)
+        df = pd.DataFrame(result)
+        log.info(f"result: {df}")
+        # save result to parquet
+        file_name = "/tmp/ci_logs/single_request_result.parquet"
+        Path(file_name).parent.mkdir(parents=True, exist_ok=True)
+        df.to_parquet(file_name)
+        if is_check:
+            assert_statistic(self.health_checkers, succ_rate_threshold=0.98)
+            # get each checker's rto
+            for k, v in self.health_checkers.items():
+                log.info(f"{k} rto: {v.get_rto()}")
+                rto = v.get_rto()
+                rto_threshold = 10
+                pytest.assume(
+                    rto <= rto_threshold,
+                    f"{self.health_checkers[k].c_name} {k} rto expect {rto_threshold}s but get {rto}s, {self.health_checkers[k].fail_records}",
+                )
+
+            # if Op.insert in self.health_checkers:
+            #     # verify the no insert data loss
+            #     log.info("*********************Verify Data Completeness**********************")
+            #     self.health_checkers[Op.insert].verify_data_completeness()
+
+        #
+        for k, v in self.health_checkers.items():
+            v.reset()
+
+        # wait all pod running
+        file_path = f"{str(Path(__file__).parent.parent.parent)}/deploy/milvus_crd.yaml"
+        with open(file_path, "r") as f:
+            config = full_load(f)
+        meta_name = config["metadata"]["name"]
+        label_selector = f"app.kubernetes.io/instance={meta_name}"
+        is_ready = wait_pods_ready("chaos-testing", label_selector)
+        pytest.assume(is_ready is True, f"expect all pods ready but got {is_ready}")
+        cc.start_monitor_threads(self.health_checkers)
+        sleep(120)
+        log.info("check succ rate after rolling update finished")
+        for k, v in self.health_checkers.items():
+            v.check_result()
+        for k, v in self.health_checkers.items():
+            log.info(f"{k} failed request: {v.fail_records}")
+        for k, v in self.health_checkers.items():
+            log.info(f"{k} rto: {v.get_rto()}")
+        assert_statistic(self.health_checkers, succ_rate_threshold=1.0)
+        log.info("*********************Test Completed**********************")