Kafka MCP Server for Topics, Offsets, Partitions

Overview

Kafka MCP Server provides a lightweight Model Context Protocol (MCP) front-end for managing Kafka topics, partitions and offsets. It exposes a simple HTTP API that lets developers inspect message contents, query and modify offsets, and perform common topic/partition operations from tools, CI pipelines, or MCP-aware clients. The server is useful for debugging producers and consumers, recovering from offset issues, and integrating Kafka state into observability or AI-driven tooling.

The server is intended to sit next to your Kafka cluster and act as a controlled management surface. Rather than giving direct broker access to every tool, clients communicate with the MCP server which implements a small set of standardized operations for listing topics, reading messages, producing messages, and manipulating consumer offsets. This simplifies integration with automations and keeps operational commands audit-friendly.

Features

• List Kafka topics and inspect configuration metadata
• View partitions and partition leaders / replicas
• Read messages by partition and offset (peek / tail / ranged reads)
• Produce messages via HTTP (single or batched)
• Query consumer group offsets and commit or reset offsets
• Adjust offsets for partitions (seek to earliest/latest or specific offsets)
• Simple REST API compatible with MCP client integrations
• Environment-based configuration (Docker & Kubernetes friendly)
• Lightweight, suitable for debugging, CI, and AI integrations that need Kafka context

Installation / Configuration

Quick Docker run (example):

docker run -d \
  -p 8080:8080 \
  -e KAFKA_BOOTSTRAP_SERVERS="kafka-broker:9092" \
  -e MCP_BIND_ADDR="0.0.0.0:8080" \
  --name kafka-mcp \
  shivamxtech/kafka-mcp:latest

Environment variables (common):

• KAFKA_BOOTSTRAP_SERVERS: comma-separated bootstrap servers (required)
• MCP_BIND_ADDR: address and port to bind the HTTP server (default: 0.0.0.0:8080)
• LOG_LEVEL: debug/info/warn/error
• AUTH_TOKEN: optional token for simple API authentication

YAML configuration (example):

kafka:
  bootstrapServers: "kafka-1:9092,kafka-2:9092"
server:
  bindAddr: "0.0.0.0:8080"
auth:
  token: "s3cr3t-token"   # optional
limits:
  maxMessageBytes: 1048576

Run with a config file (example CLI):

kafka-mcp --config /etc/kafka-mcp/config.yaml

API security: for production, run the server behind an authenticated gateway or enable TLS and token-based authentication. The server is designed for operational use and should not be exposed without proper access controls.

Available Resources

Core HTTP endpoints (examples — adapt to your deployment URL prefix):

Example: list topics

curl -H "Authorization: Bearer $AUTH_TOKEN" \
  http://localhost:8080/mcp/topics

Example: read messages (partition 0, offset 10, limit 5)

curl "http://localhost:8080/mcp/topics/my-topic/messages?partition=0&offset=10&limit=5"

Example: produce messages

curl -X POST -H "Content-Type: application/json" \
  -d '{"messages":[{"key":"k1","value":"hello"},{"value":"world"}]}' \
  http://localhost:8080/mcp/topics/my-topic/messages

Available clients / integrations:

• Use any HTTP client (curl, Postman)
• Integrate with MCP-aware tooling (AI agents, observability systems)
• Example Node.js snippet:

const axios = require('axios');
const base = 'http://localhost:8080/mcp';
await axios.get(`${base}/topics`);

Use Cases

• Inspect consumer lag: fetch consumer group offsets and compare to topic end offsets to identify lagging consumers.
  • Example: GET /mcp/consumer-groups/payment-service/offsets then list latest offsets for each partition.
• Recover from bad deployments: reset offsets for a consumer group to replay messages from an earlier offset or the beginning.
  • Example: POST to /mcp/consumer-groups/{group}/offsets with desired partition -> offset map.
• Debug message content: quickly peek at messages in a partition to validate schema, debug malformed payloads, or inspect headers.
  • Example: GET /mcp/topics/orders/messages?partition=2&offset=12345&limit=10
• CI / testing: create short-lived topics, produce deterministic messages, and run tests that assert consumer behavior without giving test runners raw Kafka access.
• AI/automation integration: supply Kafka context (topic metadata, recent messages, offsets) to automated agents or tools that need to reason about streaming data as part of a workflow.

Tips and Best Practices

• Limit access: restrict the MCP server to trusted networks or enable auth tokens and TLS.
• Use read-only mode for diagnostics where possible to prevent accidental offset changes.
• Set sensible message read limits to avoid large responses; use pagination for bulk inspection.
• Prefer using the server behind an API gateway when integrating into production automation workflows.

For full API details and examples, consult the repository or the project’s API spec/Swagger (if provided) in your deployment.