Patronus AI MCP Server: Agent & RAG Optimization

Overview

The Patronus AI MCP Server implements a Model Context Protocol (MCP) runtime intended for developing, testing, and optimizing AI agents and Retrieval-Augmented Generation (RAG) applications. It provides a lightweight API surface to simulate agent interactions, exercise retrieval pipelines, and benchmark end-to-end workflows against configurable models, retrievers, and memory layers. The server is designed to help engineering teams iterate on prompt/context management, routing logic, and fault-handling before deploying to production.

By centralizing common operations—context window management, request shaping, retries, and telemetry—the MCP server simplifies reproducible experiments and load testing. It can be run locally, in containers, or as part of CI workflows to validate performance, reliability, and correctness of agent behaviors and RAG chains.

Features

• Simple HTTP API for running agent simulations and RAG evaluations.
• Config-driven model and vector store adapters (pluggable backends).
• Context management utilities (truncation, summarization, memory injection).
• Batch and single-request modes for throughput and latency testing.
• Built-in metrics and health endpoints for observability.
• Configurable retry and timeout policies for robust testing.
• Support for synthetic test documents and conversation traces.
• Docker-friendly for local dev and CI integration.

Installation / Configuration

Clone and run locally:

git clone https://github.com/patronus-ai/patronus-mcp-server.git
cd patronus-mcp-server
# Optionally create a virtualenv
python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
python -m patronus_mcp_server.main --config config/local.yaml

Run with Docker:

docker build -t patronus-mcp-server:latest .
docker run -p 8080:8080 \
  -e MCP_SERVER_HOST=0.0.0.0 \
  -e CONFIG_PATH=/app/config/local.yaml \
  patronus-mcp-server:latest

Quick docker-compose snippet:

version: "3.8"
services:
  mcp:
    image: patronus-mcp-server:latest
    ports:
      - "8080:8080"
    environment:
      - CONFIG_PATH=/app/config/local.yaml
      - MODEL_BACKEND=http://local-model:8000
      - VECTOR_STORE_URL=http://local-vector:7700
    volumes:
      - ./config:/app/config

Common environment variables

• MCP_SERVER_HOST — bind address (default 127.0.0.1)
• MCP_SERVER_PORT — port (default 8080)
• CONFIG_PATH — path to YAML config
• MODEL_BACKEND — URL of model inference service
• VECTOR_STORE_URL — URL of vector database
• LOG_LEVEL — debug/info/warn

Example config (config/local.yaml)

server:
  host: 0.0.0.0
  port: 8080

model_backend:
  url: http://localhost:8000
  default_model: llama-2-13b
  max_tokens: 1024

vector_store:
  url: http://localhost:7700
  index: documents

agent_defaults:
  context_window: 2048
  summary_threshold: 1500
  retry_policy:
    max_attempts: 3
    backoff_seconds: 1.5

Available Resources

• HTTP API endpoints (see table below) for common operations.
• OpenAPI / Swagger UI (when enabled) for interactive exploration.
• Example configs and synthetic datasets in the repo to run repeatable tests.
• Metrics exposed via a /metrics endpoint (Prometheus compatible) to monitor latency, success rates, and resource usage.
• Logs and tracing integrations (OTel) when configured through environment variables.

Endpoints overview

Example: run an agent scenario

curl -X POST http://localhost:8080/agent/run \
  -H "Content-Type: application/json" \
  -d '{
    "agent_id": "support-bot",
    "input": "Customer: My order never arrived. Can you help?",
    "context": {"user_id": "u-1234"},
    "mode": "single"
  }'

Example: RAG evaluation

curl -X POST http://localhost:8080/rag/evaluate \
  -H "Content-Type: application/json" \
  -d '{
    "query": "How to reset my password?",
    "documents": ["doc1 text...", "doc2 text..."],
    "retrieval_top_k": 5,
    "evaluate": true
  }'

Use Cases

• Agent prompt and context optimization: iterate on prompt templates and context injection rules to see how different truncation strategies and summaries affect reply quality and token usage.
• RAG QA evaluation: compare retrieval strategies (BM25 vs vector similarity) and measure downstream answer correctness and hallucination rates across datasets.
• Latency and throughput testing: run batch agent scenarios to simulate expected request patterns and identify bottlenecks in model backends or vector stores.
• Fault injection and resilience testing: exercise retry policies, timeouts, and corrupt memory states to validate agent behavior under partial failures.
• CI gating: include deterministic MCP server runs in your CI to assert that new code doesn’t degrade agent behavior metrics (accuracy, latency, token usage).

Getting help

• Repository: https://github.com/patronus-ai/patronus-mcp-server
• Open an issue on GitHub for bugs or feature requests.
• Use the included examples and configs as a starting point; adapt model and vector backend URLs to your environment.