Overview

This project implements a high-performance Model Context Protocol (MCP) server for Apache Kafka, written in Go and using franz-go as the Kafka client library. The server mediates model context and streaming interactions by mapping MCP-style requests/responses onto Kafka topics, enabling low-latency, durable, and horizontally scalable model context delivery for AI/ML systems.

Designed for production usage, the server focuses on fast message I/O, efficient batching, and robust Kafka client configuration (SASL/TLS, consumer groups, timeouts). It is useful when you need to stream context to or from large language models, coordinate state across distributed inference workers, or persist/replicate model context via Kafka topics while keeping latency low.

Features

Built in Go for performance and easy deployment
franz-go Kafka client for modern Kafka features and throughput
Mapping of MCP requests/responses onto Kafka topics for streaming model context
Consumer-group based processing for horizontal scalability
TLS and SASL support for secure Kafka clusters
Configurable batching, timeouts, and topic naming
Lightweight binary suitable for containers and Kubernetes

Installation / Configuration

Clone, build, and run the server. Replace config values with your Kafka cluster details.

Basic steps:

git clone https://github.com/tuannvm/kafka-mcp-server.git
cd kafka-mcp-server
go build -o kafka-mcp-server

Run the server with a YAML config file (example below) or environment variables / CLI flags (if supported by the binary):

Example config.yaml:

kafka:
  brokers:
    - "kafka1:9092"
    - "kafka2:9092"
  client_id: "mcp-server-1"
  group_id: "mcp-server-group"
  request_topic: "mcp-requests"
  response_topic: "mcp-responses"
  enable_tls: false
  sasl:
    mechanism: "plain"     # "plain", "scram_sha256", etc.
    username: "user"
    password: "pass"

server:
  listen_addr: "0.0.0.0:8080"
  max_batch_size: 100
  read_timeout_ms: 500
  write_timeout_ms: 200

Start:

./kafka-mcp-server --config=config.yaml

Common environment override example:

export KAFKA_BROKERS="kafka1:9092,kafka2:9092"
export KAFKA_CLIENT_ID="mcp-server-1"
./kafka-mcp-server --config=config.yaml

Configuration keys (summary):

Key	Type	Description
kafka.brokers	[]string	Kafka broker addresses
kafka.client_id	string	Kafka client identifier
kafka.group_id	string	Consumer group for processing requests
kafka.request_topic	string	Topic to read MCP requests from
kafka.response_topic	string	Topic to write MCP responses to
kafka.enable_tls	bool	Toggle TLS
kafka.sasl.*	object	SASL auth settings
server.listen_addr	string	Local listen address for any control plane API
server.max_batch_size	int	Max messages to batch when producing
server.read_timeout_ms / write_timeout_ms	int	Kafka I/O timeouts

Note: Adjust topic names and consumer group to match your deployment and retention strategy. Ensure topics exist or that the Kafka cluster allows automatic topic creation.

Available Resources

GitHub repository: https://github.com/tuannvm/kafka-mcp-server
franz-go Kafka client: https://github.com/twmb/franz-go
Apache Kafka docs: https://kafka.apache.org
Example tools for testing:
- kafka-console-producer / kafka-console-consumer (Kafka CLI)
- kafkacat / kcat for producing/consuming raw messages

Quick test using Kafka CLI (produce a request):

echo '{"id":"req-123","type":"mcp_request","payload":{"prompt":"Summarize this text..."}}' \
  | kafka-console-producer.sh --broker-list localhost:9092 --topic mcp-requests

Consume responses:

kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic mcp-responses --from-beginning

Use Cases

Real-time LLM context streaming: Route user conversation history and retrieval results into inference workers via Kafka topics so models receive up-to-date context with durable storage.
Multi-tenant model coordination: Partition model context per tenant using Kafka topic keys and consumer groups to isolate workloads while sharing a single MCP server binary.
Embeddings and retrieval pipeline: Stream embeddings or retrieval metadata from preprocessors into a centralized context topic that model-serving components consume to augment prompts.
Cache warm-up and cold-start mitigation: Produce warm-up context messages to the topic to pre-fill inference caches when scaling up model replicas.
Observability and audit trails: Persist every model-context exchange for later replay, debugging, or compliance analysis, leveraging Kafka retention and compaction features.

Operational Notes

Monitor consumer lag and partition assignments to ensure low latency. Configure an appropriate consumer group size relative to the number of partitions for scalability.
Tune batch sizes and timeouts based on message size and latency requirements—smaller batches for lower latency, larger batches for throughput.
Secure your Kafka cluster with TLS and SASL when operating in production; confirm franz-go settings match broker authentication.
Test topic retention, compaction, and partitioning

Kafka MCP Server Built in Go with franz-go