wehub-resource-sync
498b235461
3.2 KiB
Milvus Streaming System
How to use this knowledge base: This README provides the architecture overview of the WAL system. Each component name is a link to its detailed doc. When your task involves a specific component, read the linked doc to get implementation details, interfaces, and code locations before making changes.
Architecture Overview
Milvus uses a log-structured WAL (Write-Ahead Log) as its single source of truth for all data mutations and metadata changes. The WAL spans multiple PChannels distributed across StreamingNodes, coordinated by StreamingCoord, and accessed by other components through a StreamingClient library.
Channel Model
The WAL is partitioned into PChannels (physical), mapped to VChannels (logical, per-shard-of-collection), with a singleton CChannel (control) for cluster-wide ordering. See channel/channel.md for details.
Message Model
Every WAL entry is a Message representing a system event, with TimeTick as PChannel-level monotonically increasing log sequence number.
Data Flow
- DML (Insert/Delete, etc.): Client → Proxy → StreamingClient.Append → StreamingNode → WAL Backend.
- DDL/DCL (CreateCollection, RBAC, etc.): Client → Proxy → StreamingClient.Broadcast → StreamingCoord.Broadcaster → StreamingNodes (all relevant PChannels atomically) → WAL Backend.
- WALInternal (TimeTick, Flush, CreateSegment, Txn): Self-generated by WAL System, not from external clients.
- Replicated: Primary WAL → CDC ChannelReplicator → Secondary Proxy → Secondary WAL (Replicate Interceptor) → WAL Backend.
- Consume & Persist: WAL Backend → RecoveryStorage (checkpoint, metadata, segment data persistence) + Broadcaster ACK (confirms per-PChannel broadcast delivery back to StreamingCoord).
Components
StreamingCoord (singleton, runs inside RootCoord process):
- Channel Management: PChannel-to-StreamingNode assignment, node health monitoring, VChannel/CChannel allocation.
- Broadcaster: Cross-PChannel atomic broadcast (DDL/DCL) with resource locking and ACK tracking.
StreamingNode (multiple instances, each manages a subset of PChannels):
- TimeTick & Transaction: TimeTick allocation/confirmation, transaction lifecycle, LastConfirmedMessageID.
- Lock: Exclusive/shared append access at VChannel or PChannel scope.
- Shard Management: Per-PChannel collection/partition/segment metadata and segment assignment.
- RecoveryStorage: Checkpoint, metadata and data persistence and WAL-based state recovery.
StreamingClient: In-process Append/Read/Broadcast API with service discovery and auto-reconnect.
Replication & CDC: Star-topology cross-cluster WAL replication and role management.
WAL Backend: Durable storage layer (Kafka/Pulsar/Woodpecker/RMQ), one topic per PChannel.