chore: import upstream snapshot with attribution

registry/etcd/PERFORMANCE.md

@@ -0,0 +1,160 @@
+# Etcd Registry Performance Improvements
+
+This document describes the improvements made to address etcd authentication performance issues and cache penetration problems.
+
+## Problem Statement
+
+### Background
+When etcd server authentication is enabled, a serious performance bottleneck can occur at scale. This was observed in production environments with 4000+ service pods.
+
+### Issues Identified
+
+#### 1. High Authentication QPS
+- **Root Cause**: The etcd registry used `KeepAliveOnce` for lease renewal, which requires a new authentication request for each call
+- **Impact**: With 4000+ pods registering every 30s (default RegisterInterval), this creates ~110 QPS of authentication requests
+- **Limitation**: A typical 3-node etcd cluster (64C 256G HDD) can only handle ~100 QPS for authentication
+- **Result**: Authentication requests overwhelm etcd, causing KeepAlive failures and service deregistrations
+
+#### 2. Cache Penetration
+- **Trigger**: When KeepAlive fails, services deregister from etcd
+- **Chain Reaction**: 
+  1. Registry watcher detects deletions
+  2. Cache is cleared based on delete events
+  3. All subsequent service lookups hit etcd directly (cache miss)
+  4. Etcd is already overloaded, causing more failures
+- **Result**: Cascading failure where all gRPC requests fail
+
+## Solution
+
+### 1. Use Long-Lived KeepAlive Channels
+
+**Change**: Replaced `KeepAliveOnce` with `KeepAlive`
+
+**Implementation**:
+- Added keepalive channel management to `etcdRegistry` struct
+- Created `startKeepAlive()` method that establishes a long-lived keepalive stream
+- Modified `registerNode()` to reuse existing keepalive channels
+- Added `stopKeepAlive()` for proper cleanup on deregistration
+
+**Benefits**:
+- **97% reduction in auth requests**: From ~110 QPS to ~3-4 QPS (4000 pods / TTL period)
+- **Single authentication per lease**: KeepAlive authenticates once when establishing the stream
+- **Automatic renewal**: Etcd sends keepalive responses automatically through the channel
+
+**Code Changes**:
+```go
+// Before: New auth request every heartbeat
+if _, err := e.client.KeepAliveOnce(context.TODO(), leaseID); err != nil {
+    // handle error
+}
+
+// After: Single auth request, reused channel
+if err := e.startKeepAlive(s.Name+node.Id, leaseID); err != nil {
+    // handle error
+}
+```
+
+### 2. Verify Cache Penetration Protection
+
+**Existing Protection**: The registry cache already uses `singleflight` pattern to prevent stampede
+
+**How it Works**:
+- When cache expires/is empty, first request triggers etcd query
+- Concurrent requests for same service wait for the first request to complete
+- All waiting requests receive the same result
+- Only ONE etcd query happens regardless of concurrent request count
+
+**Additional Safety**:
+- Stale cache is returned when etcd fails (if cache data exists)
+- Prevents cascading failures by avoiding repeated failed requests to etcd
+
+**Verification**:
+Added comprehensive tests to confirm this behavior works correctly under load.
+
+## Performance Impact
+
+### Authentication Load Reduction
+- **Before**: 4000 pods × (1 auth / 30s) = ~133 auth/sec
+- **After**: 4000 pods × (1 auth / lease_ttl) ≈ 3-4 auth/sec (assuming 15min lease TTL)
+- **Reduction**: ~97%
+
+### Cache Penetration Prevention
+- **Before**: When cache clears, 1000s of concurrent requests → 1000s of etcd queries
+- **After**: When cache clears, 1000s of concurrent requests → 1 etcd query (singleflight)
+- **Reduction**: ~99.9%
+
+## Testing
+
+### Unit Tests
+1. **TestKeepAliveManagement**: Validates keepalive lifecycle
+   - Verifies channels are created on registration
+   - Confirms channels are cleaned up on deregistration
+   
+2. **TestKeepAliveReducesAuthRequests**: Confirms channel reuse
+   - Multiple re-registrations use the same keepalive channel
+   - Validates auth request reduction
+
+3. **TestKeepAliveChannelReconnection**: Tests error handling
+   - Verifies proper cleanup when keepalive channel closes
+
+4. **TestSingleflightPreventsStampede**: Validates cache behavior
+   - 10 concurrent requests → 1 etcd query
+   
+5. **TestStaleCacheOnError**: Confirms graceful degradation
+   - Returns stale cache when etcd fails
+
+6. **TestCachePenetrationPrevention**: End-to-end validation
+   - 50 concurrent requests during etcd failure → 1 etcd query
+   - All requests receive stale cache
+
+### Integration Tests
+- CI workflow runs tests against real etcd instance
+- Validates behavior with actual etcd keepalive channels
+- Tests run with race detector enabled
+
+## Migration Guide
+
+### For Library Users
+No code changes required! The improvements are transparent:
+- Existing applications automatically benefit from reduced auth load
+- No API changes to `registry.Registry` interface
+
+### For Plugin Developers
+If you maintain a custom registry plugin:
+- Consider implementing long-lived keepalive channels
+- Ensure your cache implementation uses singleflight pattern
+- Add tests for concurrent access patterns
+
+## Monitoring Recommendations
+
+### Key Metrics to Track
+1. **Etcd Authentication Rate**: Should drop by ~97%
+2. **Etcd Query Rate**: Monitor for stampede prevention
+3. **Service Registration Success Rate**: Should improve under load
+4. **Cache Hit Rate**: Should remain high even during etcd issues
+
+### Expected Behavior
+- **Normal Operation**: Low auth QPS, high cache hit rate
+- **During Etcd Issues**: Stale cache served, limited etcd queries
+- **After Recovery**: Cache refreshes gradually, no stampede
+
+## Related Issues
+
+- Original Issue: [BUG] etcd authentication performance issue and registry cache penetration
+- Etcd Documentation: https://etcd.io/docs/latest/learning/api/#lease-keepalive
+- Singleflight Pattern: https://pkg.go.dev/golang.org/x/sync/singleflight
+
+## Security Considerations
+
+- **No Authentication Bypass**: Changes only reduce frequency, not security
+- **Proper Cleanup**: Keepalive channels properly closed on deregistration
+- **Race Condition Free**: All map operations properly synchronized
+- **No Resource Leaks**: Goroutines terminate when channels close
+
+## Future Enhancements
+
+Potential improvements for consideration:
+1. **Adaptive TTL**: Adjust keepalive frequency based on load
+2. **Circuit Breaker**: Temporarily stop queries when etcd is degraded
+3. **Metrics**: Expose keepalive channel count, auth rate, etc.
+4. **Backoff**: Exponential backoff on keepalive failures

registry/etcd/etcd.go

@@ -0,0 +1,532 @@
+// Package etcd provides an etcd service registry
+package etcd
+
+import (
+	"context"
+	"encoding/json"
+	"errors"
+	"net"
+	"os"
+	"path"
+	"sort"
+	"strings"
+	"sync"
+	"time"
+
+	hash "github.com/mitchellh/hashstructure"
+	mtls "go-micro.dev/v6/internal/util/tls"
+	"go-micro.dev/v6/logger"
+	"go-micro.dev/v6/registry"
+	"go.etcd.io/etcd/api/v3/v3rpc/rpctypes"
+	clientv3 "go.etcd.io/etcd/client/v3"
+	"go.uber.org/zap"
+)
+
+var (
+	prefix = "/micro/registry/"
+)
+
+type etcdRegistry struct {
+	client  *clientv3.Client
+	options registry.Options
+
+	sync.RWMutex
+	register      map[string]uint64
+	leases        map[string]clientv3.LeaseID
+	keepaliveChs  map[string]<-chan *clientv3.LeaseKeepAliveResponse
+	keepaliveStop map[string]chan bool
+}
+
+func NewEtcdRegistry(opts ...registry.Option) registry.Registry {
+	e := &etcdRegistry{
+		options:       registry.Options{},
+		register:      make(map[string]uint64),
+		leases:        make(map[string]clientv3.LeaseID),
+		keepaliveChs:  make(map[string]<-chan *clientv3.LeaseKeepAliveResponse),
+		keepaliveStop: make(map[string]chan bool),
+	}
+	username, password := os.Getenv("ETCD_USERNAME"), os.Getenv("ETCD_PASSWORD")
+	if len(username) > 0 && len(password) > 0 {
+		opts = append(opts, Auth(username, password))
+	}
+	address := os.Getenv("MICRO_REGISTRY_ADDRESS")
+	if len(address) > 0 {
+		opts = append(opts, registry.Addrs(address))
+	}
+	_ = configure(e, opts...)
+	return e
+}
+
+func configure(e *etcdRegistry, opts ...registry.Option) error {
+	config := clientv3.Config{
+		Endpoints: []string{"127.0.0.1:2379"},
+	}
+
+	for _, o := range opts {
+		o(&e.options)
+	}
+
+	if e.options.Timeout == 0 {
+		e.options.Timeout = 5 * time.Second
+	}
+
+	if e.options.Logger == nil {
+		e.options.Logger = logger.DefaultLogger
+	}
+
+	config.DialTimeout = e.options.Timeout
+
+	if e.options.Secure || e.options.TLSConfig != nil {
+		tlsConfig := e.options.TLSConfig
+		if tlsConfig == nil {
+			// Use environment-based config - secure by default
+			tlsConfig = mtls.Config()
+		}
+
+		config.TLS = tlsConfig
+	}
+
+	if e.options.Context != nil {
+		u, ok := e.options.Context.Value(authKey{}).(*authCreds)
+		if ok {
+			config.Username = u.Username
+			config.Password = u.Password
+		}
+		cfg, ok := e.options.Context.Value(logConfigKey{}).(*zap.Config)
+		if ok && cfg != nil {
+			config.LogConfig = cfg
+		}
+	}
+
+	var cAddrs []string
+
+	for _, address := range e.options.Addrs {
+		if len(address) == 0 {
+			continue
+		}
+		addr, port, err := net.SplitHostPort(address)
+		if ae, ok := err.(*net.AddrError); ok && ae.Err == "missing port in address" {
+			port = "2379"
+			addr = address
+			cAddrs = append(cAddrs, net.JoinHostPort(addr, port))
+		} else if err == nil {
+			cAddrs = append(cAddrs, net.JoinHostPort(addr, port))
+		}
+	}
+
+	// if we got addrs then we'll update
+	if len(cAddrs) > 0 {
+		config.Endpoints = cAddrs
+	}
+
+	cli, err := clientv3.New(config)
+	if err != nil {
+		return err
+	}
+	e.client = cli
+	return nil
+}
+
+func encode(s *registry.Service) string {
+	b, _ := json.Marshal(s)
+	return string(b)
+}
+
+func decode(ds []byte) *registry.Service {
+	var s *registry.Service
+	_ = json.Unmarshal(ds, &s)
+	return s
+}
+
+func nodePath(s, id string) string {
+	service := strings.ReplaceAll(s, "/", "-")
+	node := strings.ReplaceAll(id, "/", "-")
+	return path.Join(prefix, service, node)
+}
+
+func servicePath(s string) string {
+	return path.Join(prefix, strings.ReplaceAll(s, "/", "-"))
+}
+
+func (e *etcdRegistry) Init(opts ...registry.Option) error {
+	return configure(e, opts...)
+}
+
+func (e *etcdRegistry) Options() registry.Options {
+	return e.options
+}
+
+func (e *etcdRegistry) registerNode(s *registry.Service, node *registry.Node, opts ...registry.RegisterOption) error {
+	if len(s.Nodes) == 0 {
+		return errors.New("require at least one node")
+	}
+
+	// check existing lease cache
+	e.RLock()
+	leaseID, ok := e.leases[s.Name+node.Id]
+	e.RUnlock()
+
+	log := e.options.Logger
+
+	if !ok {
+		// missing lease, check if the key exists
+		ctx, cancel := context.WithTimeout(context.Background(), e.options.Timeout)
+		defer cancel()
+
+		// look for the existing key
+		rsp, err := e.client.Get(ctx, nodePath(s.Name, node.Id), clientv3.WithSerializable())
+		if err != nil {
+			return err
+		}
+
+		// get the existing lease
+		for _, kv := range rsp.Kvs {
+			if kv.Lease > 0 {
+				leaseID = clientv3.LeaseID(kv.Lease)
+
+				// decode the existing node
+				srv := decode(kv.Value)
+				if srv == nil || len(srv.Nodes) == 0 {
+					continue
+				}
+
+				// create hash of service; uint64
+				h, err := hash.Hash(srv.Nodes[0], nil)
+				if err != nil {
+					continue
+				}
+
+				// save the info
+				e.Lock()
+				e.leases[s.Name+node.Id] = leaseID
+				e.register[s.Name+node.Id] = h
+				e.Unlock()
+
+				break
+			}
+		}
+	}
+
+	var leaseNotFound bool
+
+	// renew the lease if it exists
+	if leaseID > 0 {
+		log.Logf(logger.TraceLevel, "Renewing existing lease for %s %d", s.Name, leaseID)
+
+		// Start long-lived keepalive channel to reduce auth requests
+		// startKeepAlive checks if already running and is atomic
+		if err := e.startKeepAlive(s.Name+node.Id, leaseID); err != nil {
+			if err != rpctypes.ErrLeaseNotFound {
+				return err
+			}
+
+			log.Logf(logger.TraceLevel, "Lease not found for %s %d", s.Name, leaseID)
+			// lease not found do register
+			leaseNotFound = true
+		}
+	}
+
+	// create hash of service; uint64
+	h, err := hash.Hash(node, nil)
+	if err != nil {
+		return err
+	}
+
+	// get existing hash for the service node
+	e.Lock()
+	v, ok := e.register[s.Name+node.Id]
+	e.Unlock()
+
+	// the service is unchanged, skip registering
+	if ok && v == h && !leaseNotFound {
+		log.Logf(logger.TraceLevel, "Service %s node %s unchanged skipping registration", s.Name, node.Id)
+		return nil
+	}
+
+	service := &registry.Service{
+		Name:      s.Name,
+		Version:   s.Version,
+		Metadata:  s.Metadata,
+		Endpoints: s.Endpoints,
+		Nodes:     []*registry.Node{node},
+	}
+
+	var options registry.RegisterOptions
+	for _, o := range opts {
+		o(&options)
+	}
+
+	ctx, cancel := context.WithTimeout(context.Background(), e.options.Timeout)
+	defer cancel()
+
+	var lgr *clientv3.LeaseGrantResponse
+	if options.TTL.Seconds() > 0 {
+		// get a lease used to expire keys since we have a ttl
+		lgr, err = e.client.Grant(ctx, int64(options.TTL.Seconds()))
+		if err != nil {
+			return err
+		}
+	}
+
+	// create an entry for the node
+	if lgr != nil {
+		log.Logf(logger.TraceLevel, "Registering %s id %s with lease %v and leaseID %v and ttl %v", service.Name, node.Id, lgr, lgr.ID, options.TTL)
+		_, err = e.client.Put(ctx, nodePath(service.Name, node.Id), encode(service), clientv3.WithLease(lgr.ID))
+	} else {
+		log.Logf(logger.TraceLevel, "Registering %s id %s ttl %v", service.Name, node.Id, options.TTL)
+		_, err = e.client.Put(ctx, nodePath(service.Name, node.Id), encode(service))
+	}
+	if err != nil {
+		return err
+	}
+
+	e.Lock()
+	// save our hash of the service
+	e.register[s.Name+node.Id] = h
+	// save our leaseID of the service
+	if lgr != nil {
+		e.leases[s.Name+node.Id] = lgr.ID
+	}
+	e.Unlock()
+
+	// start keepalive for the new lease
+	if lgr != nil {
+		if err := e.startKeepAlive(s.Name+node.Id, lgr.ID); err != nil {
+			log.Logf(logger.WarnLevel, "Failed to start keepalive for %s %s: %v", s.Name, node.Id, err)
+		}
+	}
+
+	return nil
+}
+
+func (e *etcdRegistry) Deregister(s *registry.Service, opts ...registry.DeregisterOption) error {
+	if len(s.Nodes) == 0 {
+		return errors.New("require at least one node")
+	}
+
+	for _, node := range s.Nodes {
+		key := s.Name + node.Id
+
+		e.Lock()
+		// delete our hash of the service
+		delete(e.register, key)
+		// delete our lease of the service
+		delete(e.leases, key)
+		e.Unlock()
+
+		// stop keepalive goroutine
+		e.stopKeepAlive(key)
+
+		ctx, cancel := context.WithTimeout(context.Background(), e.options.Timeout)
+		defer cancel()
+
+		e.options.Logger.Logf(logger.TraceLevel, "Deregistering %s id %s", s.Name, node.Id)
+		_, err := e.client.Delete(ctx, nodePath(s.Name, node.Id))
+		if err != nil {
+			return err
+		}
+	}
+
+	return nil
+}
+
+func (e *etcdRegistry) Register(s *registry.Service, opts ...registry.RegisterOption) error {
+	if len(s.Nodes) == 0 {
+		return errors.New("require at least one node")
+	}
+
+	var gerr error
+
+	// register each node individually
+	for _, node := range s.Nodes {
+		err := e.registerNode(s, node, opts...)
+		if err != nil {
+			gerr = err
+		}
+	}
+
+	return gerr
+}
+
+func (e *etcdRegistry) GetService(name string, opts ...registry.GetOption) ([]*registry.Service, error) {
+	ctx, cancel := context.WithTimeout(context.Background(), e.options.Timeout)
+	defer cancel()
+
+	rsp, err := e.client.Get(ctx, servicePath(name)+"/", clientv3.WithPrefix(), clientv3.WithSerializable())
+	if err != nil {
+		return nil, err
+	}
+
+	if len(rsp.Kvs) == 0 {
+		return nil, registry.ErrNotFound
+	}
+
+	serviceMap := map[string]*registry.Service{}
+
+	for _, n := range rsp.Kvs {
+		if sn := decode(n.Value); sn != nil {
+			s, ok := serviceMap[sn.Version]
+			if !ok {
+				s = &registry.Service{
+					Name:      sn.Name,
+					Version:   sn.Version,
+					Metadata:  sn.Metadata,
+					Endpoints: sn.Endpoints,
+				}
+				serviceMap[s.Version] = s
+			}
+
+			s.Nodes = append(s.Nodes, sn.Nodes...)
+		}
+	}
+
+	services := make([]*registry.Service, 0, len(serviceMap))
+	for _, service := range serviceMap {
+		services = append(services, service)
+	}
+
+	return services, nil
+}
+
+func (e *etcdRegistry) ListServices(opts ...registry.ListOption) ([]*registry.Service, error) {
+	versions := make(map[string]*registry.Service)
+
+	ctx, cancel := context.WithTimeout(context.Background(), e.options.Timeout)
+	defer cancel()
+
+	rsp, err := e.client.Get(ctx, prefix, clientv3.WithPrefix(), clientv3.WithSerializable())
+	if err != nil {
+		return nil, err
+	}
+
+	if len(rsp.Kvs) == 0 {
+		return []*registry.Service{}, nil
+	}
+
+	for _, n := range rsp.Kvs {
+		sn := decode(n.Value)
+		if sn == nil {
+			continue
+		}
+		v, ok := versions[sn.Name+sn.Version]
+		if !ok {
+			versions[sn.Name+sn.Version] = sn
+			continue
+		}
+		// append to service:version nodes
+		v.Nodes = append(v.Nodes, sn.Nodes...)
+	}
+
+	services := make([]*registry.Service, 0, len(versions))
+	for _, service := range versions {
+		services = append(services, service)
+	}
+
+	// sort the services
+	sort.Slice(services, func(i, j int) bool { return services[i].Name < services[j].Name })
+
+	return services, nil
+}
+
+func (e *etcdRegistry) Watch(opts ...registry.WatchOption) (registry.Watcher, error) {
+	return newEtcdWatcher(e, e.options.Timeout, opts...)
+}
+
+func (e *etcdRegistry) String() string {
+	return "etcd"
+}
+
+// startKeepAlive starts a keepalive goroutine for the given lease
+// It uses a long-lived KeepAlive channel instead of KeepAliveOnce to reduce authentication requests
+func (e *etcdRegistry) startKeepAlive(key string, leaseID clientv3.LeaseID) error {
+	e.Lock()
+	defer e.Unlock()
+
+	// check if keepalive is already running
+	if _, ok := e.keepaliveChs[key]; ok {
+		return nil
+	}
+
+	// create keepalive channel
+	ch, err := e.client.KeepAlive(context.Background(), leaseID)
+	if err != nil {
+		return err
+	}
+
+	e.keepaliveChs[key] = ch
+	stopCh := make(chan bool, 1)
+	e.keepaliveStop[key] = stopCh
+
+	// start goroutine to consume keepalive responses
+	go e.keepAliveLoop(key, ch, stopCh)
+
+	return nil
+}
+
+// keepAliveLoop consumes keepalive responses for a lease. It exits and
+// drops the cached lease/registration (via handleKeepAliveClosed) when
+// the keepalive channel closes OR a response reports a non-positive TTL.
+// Both mean the lease is gone — for example a network partition that
+// outlasted the TTL — so the next Register must re-create it rather than
+// be short-circuited by the "unchanged" check in registerNode. Without
+// the TTL guard a silently-expired lease (where the channel does not
+// promptly close) would leave the service de-registered from etcd while
+// the cache still believed it was registered, and nothing would repair it.
+func (e *etcdRegistry) keepAliveLoop(key string, ch <-chan *clientv3.LeaseKeepAliveResponse, stopCh chan bool) {
+	log := e.options.Logger
+	for {
+		select {
+		case <-stopCh:
+			log.Logf(logger.TraceLevel, "Stopping keepalive for %s", key)
+			return
+		case ka, ok := <-ch:
+			if !ok {
+				log.Logf(logger.DebugLevel, "Keepalive channel closed for %s", key)
+				e.handleKeepAliveClosed(key)
+				return
+			}
+			if ka == nil {
+				log.Logf(logger.WarnLevel, "Keepalive response is nil for %s", key)
+				continue
+			}
+			if ka.TTL <= 0 {
+				log.Logf(logger.WarnLevel, "Keepalive TTL expired for %s lease %d; dropping lease to force re-register", key, ka.ID)
+				e.handleKeepAliveClosed(key)
+				return
+			}
+			log.Logf(logger.TraceLevel, "Keepalive response for %s lease %d, TTL %d", key, ka.ID, ka.TTL)
+		}
+	}
+}
+
+// handleKeepAliveClosed is invoked by the keepalive goroutine when the
+// etcd client closes the KeepAlive channel (for example because the lease
+// expired on the server side during a network partition). In addition to
+// removing the channel bookkeeping, it drops the cached lease and
+// registration hash so the next registerNode() performs a full
+// Grant+Put re-registration instead of being short-circuited by the
+// "unchanged" check at the top of registerNode.
+func (e *etcdRegistry) handleKeepAliveClosed(key string) {
+	e.Lock()
+	defer e.Unlock()
+
+	// Only delete if still present to avoid racing with stopKeepAlive.
+	if _, exists := e.keepaliveChs[key]; exists {
+		delete(e.keepaliveChs, key)
+		delete(e.keepaliveStop, key)
+	}
+	delete(e.leases, key)
+	delete(e.register, key)
+}
+
+// stopKeepAlive stops the keepalive goroutine for the given key
+func (e *etcdRegistry) stopKeepAlive(key string) {
+	e.Lock()
+	defer e.Unlock()
+
+	if stopCh, ok := e.keepaliveStop[key]; ok {
+		close(stopCh)
+		delete(e.keepaliveChs, key)
+		delete(e.keepaliveStop, key)
+	}
+}

registry/etcd/etcd_test.go

@@ -0,0 +1,322 @@
+package etcd
+
+import (
+	"context"
+	"fmt"
+	"os"
+	"testing"
+	"time"
+
+	"go-micro.dev/v6/logger"
+	"go-micro.dev/v6/registry"
+	clientv3 "go.etcd.io/etcd/client/v3"
+)
+
+// TestKeepAliveManagement tests that keepalive channels are properly managed
+func TestKeepAliveManagement(t *testing.T) {
+	// Skip if no etcd server available
+	etcdAddr := os.Getenv("ETCD_ADDRESS")
+	if etcdAddr == "" {
+		etcdAddr = "127.0.0.1:2379"
+	}
+
+	// Try to connect to etcd
+	client, err := clientv3.New(clientv3.Config{
+		Endpoints:   []string{etcdAddr},
+		DialTimeout: 2 * time.Second,
+	})
+	if err != nil {
+		t.Skip("Etcd not available, skipping test:", err)
+		return
+	}
+	defer client.Close()
+
+	// Test connection
+	ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second)
+	defer cancel()
+	_, err = client.Get(ctx, "/test")
+	if err != nil {
+		t.Skip("Etcd not reachable, skipping test:", err)
+		return
+	}
+
+	// Create registry
+	reg := NewEtcdRegistry(
+		registry.Addrs(etcdAddr),
+		registry.Timeout(5*time.Second),
+	).(*etcdRegistry)
+
+	// Create a test service
+	service := &registry.Service{
+		Name:    "test.service",
+		Version: "1.0.0",
+		Nodes: []*registry.Node{
+			{
+				Id:      "test-node-1",
+				Address: "localhost:9090",
+			},
+		},
+	}
+
+	// Register with TTL
+	err = reg.Register(service, registry.RegisterTTL(10*time.Second))
+	if err != nil {
+		t.Fatalf("Failed to register service: %v", err)
+	}
+
+	// Wait a bit for keepalive to start
+	time.Sleep(100 * time.Millisecond)
+
+	// Check that keepalive channel was created
+	reg.RLock()
+	key := service.Name + service.Nodes[0].Id
+	_, hasKeepalive := reg.keepaliveChs[key]
+	_, hasStop := reg.keepaliveStop[key]
+	reg.RUnlock()
+
+	if !hasKeepalive {
+		t.Error("Keepalive channel was not created")
+	}
+	if !hasStop {
+		t.Error("Keepalive stop channel was not created")
+	}
+
+	// Register again (simulating re-registration)
+	// This should reuse the existing keepalive
+	err = reg.Register(service, registry.RegisterTTL(10*time.Second))
+	if err != nil {
+		t.Fatalf("Failed to re-register service: %v", err)
+	}
+
+	// Deregister
+	err = reg.Deregister(service)
+	if err != nil {
+		t.Fatalf("Failed to deregister service: %v", err)
+	}
+
+	// Wait a bit for cleanup
+	time.Sleep(100 * time.Millisecond)
+
+	// Check that keepalive was cleaned up
+	reg.RLock()
+	_, hasKeepalive = reg.keepaliveChs[key]
+	_, hasStop = reg.keepaliveStop[key]
+	reg.RUnlock()
+
+	if hasKeepalive {
+		t.Error("Keepalive channel was not cleaned up")
+	}
+	if hasStop {
+		t.Error("Keepalive stop channel was not cleaned up")
+	}
+}
+
+// TestKeepAliveReducesAuthRequests tests that KeepAlive reduces authentication requests
+// This is a conceptual test - in practice, measuring auth requests requires etcd with auth enabled
+func TestKeepAliveReducesAuthRequests(t *testing.T) {
+	// Skip if no etcd server available
+	etcdAddr := os.Getenv("ETCD_ADDRESS")
+	if etcdAddr == "" {
+		etcdAddr = "127.0.0.1:2379"
+	}
+
+	// Try to connect to etcd
+	client, err := clientv3.New(clientv3.Config{
+		Endpoints:   []string{etcdAddr},
+		DialTimeout: 2 * time.Second,
+	})
+	if err != nil {
+		t.Skip("Etcd not available, skipping test:", err)
+		return
+	}
+	defer client.Close()
+
+	// Test connection
+	ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second)
+	defer cancel()
+	_, err = client.Get(ctx, "/test")
+	if err != nil {
+		t.Skip("Etcd not reachable, skipping test:", err)
+		return
+	}
+
+	// Create registry
+	reg := NewEtcdRegistry(
+		registry.Addrs(etcdAddr),
+		registry.Timeout(5*time.Second),
+	).(*etcdRegistry)
+
+	// Create multiple test services
+	services := make([]*registry.Service, 5)
+	for i := 0; i < 5; i++ {
+		services[i] = &registry.Service{
+			Name:    fmt.Sprintf("test.service.%d", i),
+			Version: "1.0.0",
+			Nodes: []*registry.Node{
+				{
+					Id:      fmt.Sprintf("test-node-%d", i),
+					Address: fmt.Sprintf("localhost:%d", 9090+i),
+				},
+			},
+		}
+
+		// Register with TTL
+		err = reg.Register(services[i], registry.RegisterTTL(10*time.Second))
+		if err != nil {
+			t.Fatalf("Failed to register service %d: %v", i, err)
+		}
+	}
+
+	// Wait for keepalives to start
+	time.Sleep(200 * time.Millisecond)
+
+	// Verify all have keepalive channels
+	reg.RLock()
+	keepaliveCount := len(reg.keepaliveChs)
+	reg.RUnlock()
+
+	if keepaliveCount != 5 {
+		t.Errorf("Expected 5 keepalive channels, got %d", keepaliveCount)
+	}
+
+	// Simulate multiple re-registrations (heartbeats)
+	// With KeepAlive, these should NOT create new auth requests
+	for i := 0; i < 3; i++ {
+		time.Sleep(100 * time.Millisecond)
+		for _, service := range services {
+			err = reg.Register(service, registry.RegisterTTL(10*time.Second))
+			if err != nil {
+				t.Fatalf("Failed to re-register service: %v", err)
+			}
+		}
+	}
+
+	// Still should have only 5 keepalive channels (not 15 or 20)
+	reg.RLock()
+	keepaliveCount = len(reg.keepaliveChs)
+	reg.RUnlock()
+
+	if keepaliveCount != 5 {
+		t.Errorf("After re-registrations, expected 5 keepalive channels, got %d", keepaliveCount)
+	}
+
+	// Cleanup
+	for _, service := range services {
+		err = reg.Deregister(service)
+		if err != nil {
+			t.Logf("Failed to deregister service: %v", err)
+		}
+	}
+}
+
+// TestKeepAliveChannelReconnection tests that keepalive handles channel closure
+func TestKeepAliveChannelReconnection(t *testing.T) {
+	// This test verifies the goroutine properly handles channel closure
+	reg := &etcdRegistry{
+		options: registry.Options{
+			Logger: logger.DefaultLogger,
+		},
+		keepaliveChs:  make(map[string]<-chan *clientv3.LeaseKeepAliveResponse),
+		keepaliveStop: make(map[string]chan bool),
+	}
+
+	// Create a mock keepalive channel that closes immediately
+	ch := make(chan *clientv3.LeaseKeepAliveResponse)
+	close(ch)
+
+	reg.keepaliveChs["test-key"] = ch
+	stopCh := make(chan bool, 1)
+	reg.keepaliveStop["test-key"] = stopCh
+
+	// Start the goroutine manually
+	go func() {
+		log := reg.options.Logger
+		for {
+			select {
+			case <-stopCh:
+				log.Logf(logger.TraceLevel, "Stopping keepalive for test-key")
+				return
+			case ka, ok := <-ch:
+				if !ok {
+					log.Logf(logger.DebugLevel, "Keepalive channel closed for test-key")
+					reg.Lock()
+					delete(reg.keepaliveChs, "test-key")
+					delete(reg.keepaliveStop, "test-key")
+					reg.Unlock()
+					return
+				}
+				if ka == nil {
+					log.Logf(logger.WarnLevel, "Keepalive response is nil for test-key")
+					continue
+				}
+			}
+		}
+	}()
+
+	// Wait for goroutine to detect closure and cleanup
+	time.Sleep(100 * time.Millisecond)
+
+	// Verify cleanup happened
+	reg.RLock()
+	_, hasKeepalive := reg.keepaliveChs["test-key"]
+	_, hasStop := reg.keepaliveStop["test-key"]
+	reg.RUnlock()
+
+	if hasKeepalive {
+		t.Error("Keepalive channel should have been cleaned up after closure")
+	}
+	if hasStop {
+		t.Error("Stop channel should have been cleaned up after closure")
+	}
+}
+
+// TestHandleKeepAliveClosedClearsLease is a regression test for the bug
+// where a closed KeepAlive channel (e.g. because the lease expired on
+// the etcd server during a network partition) left the `leases` and
+// `register` caches populated, causing the next registerNode() to
+// short-circuit on the "unchanged" check and never re-register the
+// service. handleKeepAliveClosed must clear all four maps so that the
+// next heartbeat performs a full Grant+Put re-registration.
+func TestHandleKeepAliveClosedClearsLease(t *testing.T) {
+	reg := &etcdRegistry{
+		options:       registry.Options{Logger: logger.DefaultLogger},
+		register:      map[string]uint64{"svckey": 0xdeadbeef},
+		leases:        map[string]clientv3.LeaseID{"svckey": 42},
+		keepaliveChs:  map[string]<-chan *clientv3.LeaseKeepAliveResponse{"svckey": make(chan *clientv3.LeaseKeepAliveResponse)},
+		keepaliveStop: map[string]chan bool{"svckey": make(chan bool, 1)},
+	}
+
+	reg.handleKeepAliveClosed("svckey")
+
+	reg.RLock()
+	defer reg.RUnlock()
+
+	if _, ok := reg.keepaliveChs["svckey"]; ok {
+		t.Error("keepaliveChs not cleared")
+	}
+	if _, ok := reg.keepaliveStop["svckey"]; ok {
+		t.Error("keepaliveStop not cleared")
+	}
+	if _, ok := reg.leases["svckey"]; ok {
+		t.Error("leases not cleared — next heartbeat will skip re-registration")
+	}
+	if _, ok := reg.register["svckey"]; ok {
+		t.Error("register hash not cleared — next heartbeat will skip re-registration")
+	}
+}
+
+// TestHandleKeepAliveClosedIdempotent verifies that handleKeepAliveClosed
+// does not panic when called with a key that was already removed (which
+// can happen if stopKeepAlive/Deregister ran concurrently).
+func TestHandleKeepAliveClosedIdempotent(t *testing.T) {
+	reg := &etcdRegistry{
+		options:       registry.Options{Logger: logger.DefaultLogger},
+		register:      map[string]uint64{},
+		leases:        map[string]clientv3.LeaseID{},
+		keepaliveChs:  map[string]<-chan *clientv3.LeaseKeepAliveResponse{},
+		keepaliveStop: map[string]chan bool{},
+	}
+
+	// Should be a no-op; no panic.
+	reg.handleKeepAliveClosed("missing")
+}

registry/etcd/keepalive_test.go

@@ -0,0 +1,113 @@
+package etcd
+
+import (
+	"testing"
+	"time"
+
+	"go-micro.dev/v6/logger"
+	clientv3 "go.etcd.io/etcd/client/v3"
+)
+
+func newTestRegistry() *etcdRegistry {
+	e := &etcdRegistry{
+		register:      map[string]uint64{},
+		leases:        map[string]clientv3.LeaseID{},
+		keepaliveChs:  map[string]<-chan *clientv3.LeaseKeepAliveResponse{},
+		keepaliveStop: map[string]chan bool{},
+	}
+	e.options.Logger = logger.DefaultLogger
+	return e
+}
+
+func seedLease(e *etcdRegistry, key string, ch <-chan *clientv3.LeaseKeepAliveResponse, stop chan bool) {
+	e.Lock()
+	e.keepaliveChs[key] = ch
+	e.keepaliveStop[key] = stop
+	e.leases[key] = clientv3.LeaseID(123)
+	e.register[key] = 42
+	e.Unlock()
+}
+
+func leaseCached(e *etcdRegistry, key string) bool {
+	e.RLock()
+	defer e.RUnlock()
+	_, ok := e.leases[key]
+	return ok
+}
+
+// A keepalive response with a non-positive TTL means the lease has
+// expired server-side; the loop must drop the cached lease so the next
+// Register re-registers instead of skipping on the "unchanged" check.
+func TestKeepAliveLoopTTLExpiredDropsLease(t *testing.T) {
+	e := newTestRegistry()
+	key := "svc1node1"
+	ch := make(chan *clientv3.LeaseKeepAliveResponse, 1)
+	stop := make(chan bool, 1)
+	seedLease(e, key, ch, stop)
+
+	done := make(chan struct{})
+	go func() { e.keepAliveLoop(key, ch, stop); close(done) }()
+
+	ch <- &clientv3.LeaseKeepAliveResponse{ID: 123, TTL: 0}
+
+	select {
+	case <-done:
+	case <-time.After(2 * time.Second):
+		t.Fatal("keepAliveLoop did not return after a TTL<=0 response")
+	}
+	if leaseCached(e, key) {
+		t.Error("lease should be dropped after it expired (TTL<=0)")
+	}
+}
+
+// A closed keepalive channel also drops the cached lease.
+func TestKeepAliveLoopChannelClosedDropsLease(t *testing.T) {
+	e := newTestRegistry()
+	key := "svc1node1"
+	ch := make(chan *clientv3.LeaseKeepAliveResponse)
+	stop := make(chan bool, 1)
+	seedLease(e, key, ch, stop)
+
+	done := make(chan struct{})
+	go func() { e.keepAliveLoop(key, ch, stop); close(done) }()
+
+	close(ch)
+
+	select {
+	case <-done:
+	case <-time.After(2 * time.Second):
+		t.Fatal("keepAliveLoop did not return after the channel closed")
+	}
+	if leaseCached(e, key) {
+		t.Error("lease should be dropped after the channel closed")
+	}
+}
+
+// A healthy response must NOT drop the lease, and stopping the loop (as
+// on Deregister) leaves the cache for stopKeepAlive to clean up.
+func TestKeepAliveLoopHealthyKeepsLease(t *testing.T) {
+	e := newTestRegistry()
+	key := "svc1node1"
+	ch := make(chan *clientv3.LeaseKeepAliveResponse, 1)
+	stop := make(chan bool, 1)
+	seedLease(e, key, ch, stop)
+
+	done := make(chan struct{})
+	go func() { e.keepAliveLoop(key, ch, stop); close(done) }()
+
+	ch <- &clientv3.LeaseKeepAliveResponse{ID: 123, TTL: 30}
+	time.Sleep(50 * time.Millisecond)
+	if !leaseCached(e, key) {
+		t.Error("a healthy keepalive must not drop the lease")
+	}
+
+	stop <- true
+	select {
+	case <-done:
+	case <-time.After(2 * time.Second):
+		t.Fatal("keepAliveLoop did not return after stop")
+	}
+	if !leaseCached(e, key) {
+		t.Error("stopping the loop should not drop the lease; stopKeepAlive does that")
+	}
+}

registry/etcd/options.go

@@ -0,0 +1,37 @@
+package etcd
+
+import (
+	"context"
+
+	"go-micro.dev/v6/registry"
+	"go.uber.org/zap"
+)
+
+type authKey struct{}
+
+type logConfigKey struct{}
+
+type authCreds struct {
+	Username string
+	Password string
+}
+
+// Auth allows you to specify username/password.
+func Auth(username, password string) registry.Option {
+	return func(o *registry.Options) {
+		if o.Context == nil {
+			o.Context = context.Background()
+		}
+		o.Context = context.WithValue(o.Context, authKey{}, &authCreds{Username: username, Password: password})
+	}
+}
+
+// LogConfig allows you to set etcd log config.
+func LogConfig(config *zap.Config) registry.Option {
+	return func(o *registry.Options) {
+		if o.Context == nil {
+			o.Context = context.Background()
+		}
+		o.Context = context.WithValue(o.Context, logConfigKey{}, config)
+	}
+}

registry/etcd/watcher.go

@@ -0,0 +1,91 @@
+package etcd
+
+import (
+	"context"
+	"errors"
+	"time"
+
+	"go-micro.dev/v6/registry"
+	clientv3 "go.etcd.io/etcd/client/v3"
+)
+
+type etcdWatcher struct {
+	stop    chan bool
+	w       clientv3.WatchChan
+	client  *clientv3.Client
+	timeout time.Duration
+}
+
+func newEtcdWatcher(r *etcdRegistry, timeout time.Duration, opts ...registry.WatchOption) (registry.Watcher, error) {
+	var wo registry.WatchOptions
+	for _, o := range opts {
+		o(&wo)
+	}
+
+	ctx, cancel := context.WithCancel(context.Background())
+	stop := make(chan bool, 1)
+
+	go func() {
+		<-stop
+		cancel()
+	}()
+
+	watchPath := prefix
+	if len(wo.Service) > 0 {
+		watchPath = servicePath(wo.Service) + "/"
+	}
+
+	return &etcdWatcher{
+		stop:    stop,
+		w:       r.client.Watch(ctx, watchPath, clientv3.WithPrefix(), clientv3.WithPrevKV()),
+		client:  r.client,
+		timeout: timeout,
+	}, nil
+}
+
+func (ew *etcdWatcher) Next() (*registry.Result, error) {
+	for wresp := range ew.w {
+		if wresp.Err() != nil {
+			return nil, wresp.Err()
+		}
+		if wresp.Canceled {
+			return nil, errors.New("could not get next")
+		}
+		for _, ev := range wresp.Events {
+			service := decode(ev.Kv.Value)
+			var action string
+
+			switch ev.Type {
+			case clientv3.EventTypePut:
+				if ev.IsCreate() {
+					action = "create"
+				} else if ev.IsModify() {
+					action = "update"
+				}
+			case clientv3.EventTypeDelete:
+				action = "delete"
+
+				// get service from prevKv
+				service = decode(ev.PrevKv.Value)
+			}
+
+			if service == nil {
+				continue
+			}
+			return &registry.Result{
+				Action:  action,
+				Service: service,
+			}, nil
+		}
+	}
+	return nil, errors.New("could not get next")
+}
+
+func (ew *etcdWatcher) Stop() {
+	select {
+	case <-ew.stop:
+		return
+	default:
+		close(ew.stop)
+	}
+}