MC
Developer Tools

MCP Server: GenAI Gateway for Multi-Cluster Kubernetes

Connect GenAI systems to multiple Kubernetes clusters with the MCP server for secure, scalable multi-cluster orchestration.

View on GitHub
Quick Install
npx -y @yanmxa/multicluster-mcp-server

Overview

MCP Server is a lightweight gateway designed to connect Generative AI systems to multiple Kubernetes clusters. It provides a unified control plane for routing model inference and management requests across clusters, enabling secure, policy-driven multi-cluster orchestration for model serving and related workloads.

By centralizing routing, authentication and observability, MCP Server makes it easier to place inference workloads where they run best — on GPU pools, on-prem edge clusters, or cloud-hosted nodes — while preserving security boundaries and minimizing custom orchestration logic in GenAI applications. It’s intended for developers and infra teams that need predictable, scalable access to diverse Kubernetes clusters from a single GenAI orchestration layer.

Features

  • Cluster registration and lifecycle management (register, heartbeat, de-register)
  • Centralized routing rules for mapping models or tenants to clusters
  • Secure communication: TLS/mTLS support and bearer-token authentication
  • Pluggable authentication and policy filters (RBAC, IP allowlists)
  • Health checks, readiness probes and cluster health aggregation
  • Metrics and tracing endpoints compatible with Prometheus/OpenTelemetry
  • Connection pooling and request forwarding to reduce latency
  • Extensible plugin hooks for custom routing or admission logic
  • CLI and HTTP API for automation and CI/CD integration

Installation / Configuration

Prerequisites:

  • Kubernetes clusters you want to connect (kubeconfigs or access credentials)
  • Go toolchain (for building from source) or Docker (for running container)
  • kubectl for deploying on Kubernetes

Clone and build from source:

git clone https://github.com/yanmxa/multicluster-mcp-server.git
cd multicluster-mcp-server
go build -o mcp-server ./cmd/mcp-server
# Run locally
./mcp-server --config ./examples/config.yaml

Run as Docker container:

# Build image (if repo contains Dockerfile)
docker build -t mcp-server:local .
docker run -p 8080:8080 \
  -e MCP_LOG_LEVEL=info \
  -e MCP_CONFIG=/etc/mcp/config.yaml \
  -v $(pwd)/examples/config.yaml:/etc/mcp/config.yaml \
  mcp-server:local

Example configuration (config.yaml):

server:
  address: 0.0.0.0:8080
  tls:
    certFile: /tls/server.crt
    keyFile: /tls/server.key

auth:
  type: bearer
  token: "changeme"

clusters:
  - name: gpu-cluster
    kubeconfig: /kubeconfigs/gpu.kubeconfig
    labels:
      capability: gpu
  - name: edge-cluster
    kubeconfig: /kubeconfigs/edge.kubeconfig
    labels:
      region: edge1

routing:
  rules:
    - match:
        model: "image-*"
      target:
        labelSelector:
          capability: gpu
    - match:
        tenant: "edge-tenant"
      target:
        name: "edge-cluster"

Deploying inside Kubernetes (minimal Deployment):

apiVersion: apps/v1
kind: Deployment
metadata:
  name: mcp-server
spec:
  replicas: 2
  selector:
    matchLabels:
      app: mcp-server
  template:
    metadata:
      labels:
        app: mcp-server
    spec:
      containers:
      - name: mcp-server
        image: your-registry/mcp-server:latest
        ports:
        - containerPort: 8080
        volumeMounts:
        - name: config
          mountPath: /etc/mcp
      volumes:
      - name: config
        configMap:
          name: mcp-config

Register a cluster via HTTP API:

curl -X POST "https://mcp.example.com/v1/clusters" \
  -H "Authorization: Bearer $MCP_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"name":"edge-1","kubeconfig":"<base64-kubeconfig>","labels":{"region":"edge1"}}'

Available Tools / Resources

  • Source code and issues: https://github.com/yanmxa/multicluster-mcp-server
  • API endpoints (example):
    • GET /v1/clusters — list clusters
    • POST /v1/clusters — register cluster
    • GET /v1/health — system health
    • GET /metrics — Prometheus metrics
  • Observability: configure Prometheus to scrape /metrics and OTel for tracing
  • Extendability: plugin hooks for custom routing, auth, and admission filters

Use Cases

  • Model placement and cost optimization: Route large model inference to GPU-equipped clusters while sending lightweight models to cheaper CPU pools. Use routing rules that match on model name, tenant, or labels.
  • Data locality and compliance: Ensure inference runs in a specific region or on-prem cluster by tagging clusters with geographic labels and applying tenant-level routing policies.
  • Burst scaling and failover: Direct traffic to cloud GPU clusters during spikes, and fall back to on-prem clusters for steady-state operations. MCP Server aggregates cluster health to implement failover decisions.
  • Edge deployments: Connect many edge Kubernetes clusters and route requests from a central GenAI controller to the nearest or most appropriate edge cluster for low-latency inference.
  • Multi-team isolation: Enforce RBAC and per-tenant routing so different teams can share a single GenAI orchestration layer while keeping workloads isolated by cluster and
Tags:developer-tools