Mandoline MCP Server for AI Assistant Evaluation Framework

Overview

Mandoline MCP Server implements a Model Context Protocol (MCP) endpoint that helps teams evaluate, critique, and iteratively improve AI assistants. It provides a lightweight HTTP API and a set of pluggable evaluation tools so models (or orchestrators) can run reflection loops, produce critiques, and record evaluation artifacts in a reproducible way. The server is intended as an integration point between assistant deployments and automated evaluation workflows.

By centralizing evaluation interactions, Mandoline’s MCP server makes it easier to run repeatable test suites, capture model self-assessments, and generate structured feedback that can be used for fine-tuning or human review. Developers can run the server locally, in CI, or as part of a broader evaluation pipeline and extend it with custom evaluators, storage backends, or tooling integrations.

Features

• Implements a Model Context Protocol-style API for structured evaluation requests and responses
• HTTP/JSON API with an OpenAPI-compatible schema for integration with CI and orchestration systems
• Pluggable evaluation tools: critics, checkers, scorers, and reflection engines
• Persistence options (e.g., SQLite or PostgreSQL) for storing sessions, evaluations, and artifacts
• Container-friendly: Docker image and docker-compose examples for quick deployment
• Extensible: add custom tools, metrics, or hooks for training data generation
• Logging, metrics, and basic health endpoints for monitoring

Installation / Configuration

Below are typical ways to run the server. Replace placeholders with the specific image name, ports, or configuration values you use.

Run via Docker (quick start)

# Pull and run the server container (replace image name with the published image)
docker run -p 8080:8080 \
  -e MCP_SERVER_PORT=8080 \
  -e DATABASE_URL=sqlite:///data/mandoline.db \
  --name mandoline-mcp \
  mandoline/mandoline-mcp-server:latest

Docker Compose example

version: "3.8"
services:
  mcp-server:
    image: mandoline/mandoline-mcp-server:latest
    ports:
      - "8080:8080"
    environment:
      - MCP_SERVER_PORT=8080
      - DATABASE_URL=postgresql://user:pass@db:5432/mandoline
  db:
    image: postgres:15
    environment:
      POSTGRES_USER: user
      POSTGRES_PASSWORD: pass
      POSTGRES_DB: mandoline
    volumes:
      - db-data:/var/lib/postgresql/data
volumes:
  db-data:

Run from source (example)

git clone https://github.com/mandoline-ai/mandoline-mcp-server.git
cd mandoline-mcp-server
# install dependencies (example uses Python)
pip install -r requirements.txt
# run locally (substitute CLI or module name as provided)
python -m mandoline_mcp_server --port 8080 --db sqlite:///data/mandoline.db

Configuration (environment variables / config file)

MCP_SERVER_PORT=8080
DATABASE_URL=sqlite:///data/mandoline.db     # or postgresql://...
LOG_LEVEL=info
MAX_EVAL_WORKERS=4
ENABLE_TOOL_REGISTRY=true

Available Resources

The server exposes an HTTP API for submitting evaluation jobs, retrieving results, and registering tools. Common endpoints you’ll find (names are representative — consult the server’s OpenAPI for exact routes):

Example evaluation request (JSON)

{
  "session_id": "session-123",
  "artifact": {
    "instruction": "Summarize the document in one paragraph.",
    "context": "Long document text...",
    "model_response": "A concise summary..."
  },
  "tools": ["basic_scoring", "heuristic_critic"],
  "metadata": { "task_id": "t-42", "priority": "low" }
}

Example evaluation response (JSON)

{
  "evaluation_id": "eval-456",
  "score": 0.78,
  "critique": "Missed an important detail about X; suggested rephrase to mention Y.",
  "suggested_updates": ["Add emphasis on X", "Include explicit mention of Y"]
}

Tooling model: the server treats evaluators as “tools” that receive the same context object and return a structured result. You can implement custom tools and register them via the tools endpoint or a plugin directory.

Use Cases

1. Continuous assistant improvement

   • Automate a nightly evaluation job that runs a bank of user prompts against the current model, collects scores and critiques, and appends failed cases to a review queue or training dataset for fine-tuning.

2. Model self-reflection and iterative revision

   • Have the assistant generate an initial answer, call the MCP server to request a reflection/critique tool, and then produce an improved answer. The server