Overview

Root Signals MCP Server implements the Model Context Protocol (MCP) to expose Root Signals evaluators and judges to LLMs, agents, and other MCP clients. It lets agents ask the server to run quality checks, scoring, and LLM-as-a-judge workflows against model outputs so those outputs can be measured and iteratively improved.

The server supports both SSE (HTTP) transport for networked deployments and stdio transport for local MCP hosts. It translates Root Signals evaluators, coding-policy checks, and judge definitions into MCP tools, returning structured results (scores, feedback, and metadata) that downstream agents can use to decide next steps.

Features

Exposes Root Signals evaluators and judges as MCP tools
Supports SSE (HTTP) and stdio transports for flexible integration
Returns structured Pydantic-backed result models for predictable parsing
Tools for evaluator discovery, individual evaluations, policy adherence checks, and multi-evaluator judges (LLM-as-a-judge)
Containerized image for fast local or cloud deployment
Compatible with MCP-aware clients such as Cursor, Claude desktop, and custom agent frameworks

Installation / Configuration

Prerequisites: an API key for your Root Signals account and a host that runs Docker (or Python + dependencies if running from source).

Environment variable: ROOT_SIGNALS_API_KEY

Docker (recommended, SSE transport)

docker run \
  -e ROOT_SIGNALS_API_KEY="<your_key>" \
  -p 9090:9090 \
  --name=root-signals-mcp \
  -d ghcr.io/root-signals/root-signals-mcp:latest

Docker will spin up an HTTP SSE endpoint (default: http://0.0.0.0:9090/sse). The server also exposes a newer /mcp endpoint for MCP-compliant clients.

StdIO transport (for embedding in local MCP hosts) Configure your MCP client to launch the server as a subprocess. Example (JSON fragment for a client like Cursor):

{
  "mcpServers": {
    "root-signals": {
      "command": "python",
      "args": ["-m", "root_signals_mcp", "stdio"],
      "env": {
        "ROOT_SIGNALS_API_KEY": "<your_key>"
      }
    }
  }
}

From source (development)

git clone https://github.com/root-signals/root-signals-mcp.git
cd root-signals-mcp
pip install -r requirements.txt
export ROOT_SIGNALS_API_KEY="<your_key>"
python -m root_signals_mcp --transport sse --host 0.0.0.0 --port 9090

Available Tools

The MCP server exposes a set of tools that map to Root Signals functionality. These tools are discoverable via the MCP list_tools convention:

Tool name	Purpose
list_evaluators	Return available evaluators for the account (IDs, names, schemas)
run_evaluation	Run a single evaluator by evaluator_id against request/response pairs
run_evaluation_by_name	Run evaluator by name (useful when ID is unknown)
run_coding_policy_adherence	Evaluate code outputs against supplied policy documents (AI rules)
list_judges	List multi-evaluator judges (LLM-as-a-judge configurations)
run_judge	Execute a judge workflow that aggregates multiple evaluators

Each tool returns strongly typed JSON results including score(s), evaluator metadata, any textual feedback, and optional structured annotations that agents can parse.

Use Cases

Improve an agent’s explanation quality

A code-assistant generates an explanation; the agent calls run_evaluation to score for relevance and conciseness.
If score is below threshold, agent requests a rewrite, then re-evaluates until quality targets are met.

LLM-as-a-Judge for multi-dimensional quality control

Define a judge in Root Signals that combines correctness, safety, and style evaluators.
Agents call run_judge to receive a composite verdict and per-criterion feedback without calling multiple tools individually.

Enforce coding policies in automated code generation

Use run_coding_policy_adherence to validate that generated code follows an organization’s AI policy documents (licensing, secret handling, banned patterns).
Reject or flag outputs that fail policy checks before deployment.

Example: basic Python MCP client flow

import asyncio
from root_signals_mcp.client import RootSignalsMCPClient

async def main():
    client = RootSignalsMCPClient()
    await client.connect()  # connects to local stdio or SSE endpoint depending on config

    evaluators = await client.list_evaluators()
    print(f"Found {len(evaluators)} evaluators")

    result = await client.run_evaluation(
        evaluator_id="eval-abcdef1234",
        request="Summarize the following paragraph...",
        response="A short summary..."
    )
    print("Score:", result["score"])
    print("Feedback:", result.get("feedback"))

asyncio.run(main())

Resources & Troubleshooting

GitHub: https://github.com/root-signals/root-signals-mcp
If the server fails to fetch evaluators on startup, confirm ROOT_SIGNALS_API_KEY is set and network access to the Root Signals API is available.
For production deployments, run the SSE transport behind a reverse proxy and secure it with TLS. Limit access to the MCP endpoint and rotate API keys regularly.

This MCP server is intended for developers integrating evaluation and LLM-as-a-judge patterns into agentic AI workflows, providing repeatable, structured measurement and control for model outputs.

Root Signals MCP Server for LLM Evaluations