kubectl-mc

README

kubectl-mc

A kubectl plugin for seamless multi-cluster operations using sig-multicluster standards.

Overview

kubectl-mc (multi-cluster) provides a familiar kubectl-like user experience for managing resources across multiple Kubernetes clusters. Instead of manually switching contexts and aggregating results, users can interact with multiple clusters as naturally as they would with a single cluster.

Example

# Traditional single-cluster command
kubectl get pods

# Multi-cluster equivalent
kubectl mc get pods

The plugin automatically discovers available clusters via the hub, executes the command across all clusters using the user's own credentials, and aggregates results in the familiar kubectl output format with an additional cluster column.

Architecture

Design Principles

1. Familiar UX: Maintain kubectl's command structure and output format
2. User Credentials: Use end-user credentials, not privileged hub credentials
3. sig-multicluster Native: Leverage sig-multicluster APIs and patterns
4. Discovery-Based: Dynamically discover clusters via hub cluster
5. Vendor Neutral: Works with any sig-multicluster-compliant environment

How It Works

1. Connect to hub cluster using user's kubeconfig context
2. Discover available clusters via sig-multicluster APIs (ClusterProfile, About, or Cluster Inventory)
3. Map discovered clusters to user's kubeconfig contexts
4. Execute command on each cluster in parallel using user's credentials
5. Aggregate results and add CLUSTER column to output

For detailed architecture, see docs/architecture.md

sig-multicluster Integration

This plugin is designed for donation to the Kubernetes sig-multicluster community. The proposal has been discussed with sig-multicluster leadership, and there is interest in hosting this plugin in the sig-multicluster GitHub organization.

Key Principles

1. Vendor Neutrality: Uses only sig-multicluster standard APIs (ClusterProfile, About, or Cluster Inventory)
2. Standards Compliance: Follows sig-multicluster patterns and conventions
3. Complementary: Works alongside existing multi-cluster solutions (OCM, Rancher, etc.) without requiring them
4. User-Focused: Provides kubectl-familiar UX for day-to-day operations

Upstream Goals

This proof-of-concept is intended for:

1. Validation of the multi-cluster UX approach
2. Community feedback and iteration
3. Donation to sig-multicluster for official hosting
4. Reference implementation of user-facing sig-multicluster tooling

For design rationale and OCM relationship, see docs/design-decisions.md

Features

Phase 1: Core Functionality (Read-Only) ✅

• ✅ Hub cluster connection and authentication
• ✅ Cluster discovery via ClusterProfile API (sig-multicluster standard)
• ✅ Basic resource operations (kubectl mc get pods/deployments/services)
• ✅ Result aggregation with CLUSTER column
• ✅ Kubeconfig context mapping (manual setup via config file)
• ✅ Error handling for partial cluster failures (silent unless all fail)
• ✅ Parallel execution across clusters with concurrency control
• ✅ kubectl mc setup - Interactive cluster-to-context mapping
• ✅ kubectl mc describe <resource> - Describe resources across clusters with Events
• ✅ Dynamic column width calculation for clean table output
• ✅ Resource age calculation and display
• ✅ All-namespaces support (-A / --all-namespaces)
• ✅ Namespace default from kubeconfig context
• ✅ Cluster filtering (--clusters, --exclude)
• ✅ Wildcard cluster filtering (--clusters=prod-*, --exclude=*-staging)
• ✅ Wildcard resource name filtering (kubectl mc get pod nginx-*)
• kubectl mc logs <pod> - Get logs with cluster disambiguation

Potential Phase 1 Additions:

• kubectl mc edit <resource> - Multiplexed edit across clusters (with safety checks)

Phase 2: Automatic Credential Configuration (Planned)

Goal: Eliminate manual mapping file by using ClusterProfile properties to dynamically construct exec plugin configurations.

• ClusterProfile Properties Integration:

  • Read auth.exec.* properties from ClusterProfile resources
  • Dynamically construct exec plugin configurations on-the-fly
  • Support for AWS EKS (aws eks get-token)
  • Support for GCP GKE (gke-gcloud-auth-plugin)
  • Support for Azure AKS (kubelogin)
  • Generic exec plugin property parsing

• OCM Addon for ClusterProfile Enrichment:

  • Addon watches ManagedCluster resources
  • Automatically populates ClusterProfile properties based on cluster type
  • Detects cloud provider from ManagedCluster labels/annotations
  • Generates exec plugin metadata for each cluster

• User Credentials Model:

  • Exec plugins use user's local cloud CLI credentials
  • No secrets storage required
  • Leverages existing aws configure, gcloud auth login, az login
  • Fallback to Phase 1 manual mapping if properties not present

Example ClusterProfile with Properties:

status:
  accessProviders:
  - name: kubeconfig
    cluster:
      server: https://ABC123.eks.amazonaws.com
      certificate-authority-data: LS0t...
  properties:
  - name: auth.exec.command
    value: "aws"
  - name: auth.exec.args
    value: "eks,get-token,--cluster-name,production,--region,us-east-1"
  - name: auth.exec.apiVersion
    value: "client.authentication.k8s.io/v1beta1"
  - name: cloud.provider
    value: "aws"
  - name: cluster.type
    value: "eks"

See docs/architecture.md for detailed design.

Phase 3: Write Operations (With Safety Mechanisms)

• Write operations with safety mechanisms:
  • kubectl mc apply -f <file> - Apply with cluster selection
  • kubectl mc delete <resource> <name> - Requires confirmation prompt
  • kubectl mc patch <resource> <name> - With dry-run preview
  • kubectl mc create <resource> - With cluster targeting
• Cluster filtering for write operations (e.g., --clusters=prod-*, --exclude=staging)
• Dry-run mode for all write operations
• Progressive rollout (apply to subset, then all)
• Write operation audit logging
• Cross-cluster resource selection (labels, namespaces)
• Parallel execution with configurable concurrency

Phase 4: Enhanced UX

• Cluster health indicators in output
• Color-coded multi-cluster output
• Progress indicators for long-running operations
• Interactive cluster selection
• Resource diff across clusters

Quick Start

# 1. Build and install
make install

# 2. Enable ClusterProfile feature gate (if using OCM)
kubectl patch clustermanager cluster-manager --type=merge \
  -p '{"spec":{"registrationConfiguration":{"featureGates":[{"feature":"ClusterProfile","mode":"Enable"}]}}}'

# 3. Setup cluster-to-context mappings
kubectl mc setup --hub-context <your-hub-context>

# 4. Run multi-cluster commands
kubectl mc get pods -n default
kubectl mc get deployments --all-namespaces

Installation

From Source

# Clone the repository
git clone https://github.com/suchpuppet/kubectl-mc.git
cd kubectl-mc

# Build and install
make install

# Verify installation
kubectl plugin list | grep kubectl-mc

Manual Installation

# Build the binary
make build

# Copy to PATH
sudo cp kubectl-mc /usr/local/bin/kubectl-mc

# Or to personal bin
mkdir -p ~/bin
cp kubectl-mc ~/bin/kubectl-mc
export PATH="$HOME/bin:$PATH"

Using Go Install

# Coming soon
go install github.com/suchpuppet/kubectl-mc@latest

Usage

Prerequisites

1. Hub cluster with ClusterProfile resources (sig-multicluster standard)
2. Kubeconfig contexts for hub and spoke clusters
3. RBAC permissions to list ClusterProfiles on hub and query resources on spoke clusters

Setup Cluster Mappings

Option 1: Interactive Setup

# Discovers clusters and prompts for context names
kubectl mc setup --hub-context kind-ocm-hub

# Or use current context
kubectl config use-context kind-ocm-hub
kubectl mc setup

Option 2: Manual Configuration

Create ~/.kube/kubectl-mc-clusters.yaml:

apiVersion: kubectl-mc.k8s.io/v1alpha1
kind: ClusterMapping
hubContext: kind-ocm-hub
clusters:
- name: cluster1
  context: kind-cluster1
  namespace: open-cluster-management
- name: cluster2
  context: kind-cluster2
  namespace: open-cluster-management

Basic Commands

# Get pods across all clusters (uses namespace from kubeconfig context)
kubectl mc get pods
kubectl mc get pods -n default

# Get pods across all namespaces
kubectl mc get pods -A
kubectl mc get pods --all-namespaces

# Use wildcards in resource names
kubectl mc get pod nginx-*
kubectl mc get deployment app-???-prod
kubectl mc get service web-[0-9]*

# Get deployments
kubectl mc get deployments -n default

# Get services
kubectl mc get services -n kube-system

# Filter by cluster patterns
kubectl mc get pods --clusters=prod-*
kubectl mc get deployments --exclude=*-staging
kubectl mc get services --clusters=us-*,eu-* --exclude=*-dev

# Specify hub context explicitly
kubectl mc get pods --hub-context kind-ocm-hub -n test

# Describe resources across clusters (includes Events!)
kubectl mc describe pod nginx -n default
kubectl mc describe deployment my-app
kubectl mc describe service frontend -n production
kubectl mc describe pod nginx-* -A

Example Output (get):

$ kubectl mc get pods -n test
Discovered 2 cluster(s)
NAMESPACE   NAME                    CLUSTER      READY   STATUS    RESTARTS   AGE
test        nginx-bc7b4f464-npn2t   ocm-spoke1   1/1     Running   0          15h
test        nginx-bc7b4f464-24g52   ocm-spoke2   1/1     Running   0          2h

Example Output (describe):

$ kubectl mc describe pod nginx-bc7b4f464-npn2t -n test
Discovered 2 cluster(s)

CLUSTER: ocm-spoke1
--------------------------------------------------------------------------------
Name:         nginx-bc7b4f464-npn2t
Namespace:    test
Labels:       app=nginx
              pod-template-hash=bc7b4f464
Annotations:  <none>
Status:       Running
IP:           10.244.0.24
Containers:
  nginx:
    Image:          docker.io/bitnami/nginx:latest
    Port:           8080/TCP, 8443/TCP
    State:          Running
      Started:      Thu, 14 Nov 2024 04:05:28 +0000
    Ready:          True
    Restart Count:  0
    Environment:
      NGINX_HTTP_PORT_NUMBER:   8080
      NGINX_HTTPS_PORT_NUMBER:  8443
    Mounts:
      /opt/bitnami/nginx/conf from empty-dir (rw)
      /tmp from empty-dir (rw)
Volumes:
  empty-dir:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
Conditions:
  Type                             Status   LastTransitionTime          Reason                     Message
  ----                             ------   ------------------          ------                     -------
  PodReadyToStartContainers        True     2024-11-14T04:05:28Z                                   
  Initialized                      True     2024-11-14T04:05:28Z                                   
  Ready                            True     2024-11-14T04:05:35Z                                   
  ContainersReady                  True     2024-11-14T04:05:35Z                                   
  PodScheduled                     True     2024-11-14T04:05:28Z                                   
Events:
  Type    Reason     Age   From               Message
  ----    ------     ----  ----               -------
  Normal  Scheduled  15h   default-scheduler  Successfully assigned test/nginx-bc7b4f464-npn2t to node
  Normal  Pulled     15h   kubelet            Container image "docker.io/bitnami/nginx:latest" already present
  Normal  Created    15h   kubelet            Created container nginx
  Normal  Started    15h   kubelet            Started container nginx

Planned Commands (Not Yet Implemented)

# Logs (Phase 1)
kubectl mc logs nginx-abc123

# Write operations (Phase 3)
kubectl mc apply -f deployment.yaml --clusters=prod-*
kubectl mc delete deployment nginx --all-clusters
kubectl mc edit deployment nginx -n production

For detailed usage and examples, see docs/architecture.md

Development

Prerequisites

• Go 1.23+
• kubectl installed
• Access to a Kubernetes cluster with sig-multicluster APIs

Building

# Clone the repository
git clone https://github.com/suchpuppet/kubectl-mc.git
cd kubectl-mc

# Build the plugin
make build

# Install locally
make install

Testing

# Run unit tests
make test

# Run with coverage report
make test-coverage

# Test specific packages
go test -v ./pkg/discovery/...
go test -v ./pkg/kubeconfig/...

Current Test Coverage:

• pkg/discovery: 74.3% coverage (ClusterProfile discovery logic)
• pkg/kubeconfig: 90.5% coverage (context mapping management)
• Total: 10 tests, all passing ✅

Integration Testing:

Test against your local kind + OCM setup:

# Ensure your clusters are running
kubectl get clusterprofiles -A --context kind-ocm-hub

# Test the plugin
./kubectl-mc get pods -n test --hub-context kind-ocm-hub

Project Structure

kubectl-mc/
├── cmd/                    # CLI command implementations
├── pkg/
│   ├── discovery/         # Cluster discovery logic
│   ├── executor/          # Multi-cluster command execution
│   ├── aggregator/        # Result aggregation and formatting
│   ├── kubeconfig/        # Kubeconfig management
│   └── client/            # Kubernetes client wrappers
├── test/                  # Integration tests
└── docs/                  # Additional documentation

Contributing

This project follows sig-multicluster contribution guidelines. Contributions are welcome!

1. Fork the repository
2. Create a feature branch
3. Make your changes
4. Submit a pull request

Code Standards

• Follow standard Go conventions and idioms
• Write tests for new functionality
• Document public APIs
• Ensure sig-multicluster pattern compliance

Documentation

• Architecture - Detailed technical architecture, component design, and data flows
• Design Decisions - Rationale for key architectural and strategic decisions
• LLM Context - Comprehensive context for AI-assisted development

Resources

• Kubernetes sig-multicluster
• kubectl Plugin Development
• client-go Documentation

License

This project is governed by the Apache 2 License. See LICENSE file for details.

Status

✅ Proof of Concept - Working! ✅

The POC successfully demonstrates:

• ✅ ClusterProfile-based cluster discovery (sig-multicluster standard)
• ✅ Multi-cluster resource queries (get pods, get deployments, get services)
• ✅ Result aggregation with CLUSTER column
• ✅ Parallel execution across clusters
• ✅ Graceful error handling for partial failures
• ✅ Unit tests with 74-90% coverage on core packages
• ✅ Tested on kind + OCM setup

Current Phase: Phase 1 (Read-only operations)

Not yet ready for production use. Gathering community feedback and refining the approach.

Contact

For questions or discussions about this project, please reach out via:

• GitHub Issues
• Kubernetes sig-multicluster Slack channel
• sig-multicluster mailing list