Kiln MCP Server: RAG, Agents, Fine-Tuning

Overview

Kiln’s MCP (Model Context Protocol) server is an open-source backend that centralizes model orchestration and context management for Retrieval-Augmented Generation (RAG), agentic workflows, tool-calling, evaluation pipelines, synthetic-data generation, and fine-tuning. It exposes a stable API and connector surface so you can wire models, vector stores, and tooling into production-ready AI tasks without re-implementing orchestration primitives.

For developers this means you can run Kiln locally or in your cloud environment to:

• combine external models and private knowledge bases for RAG,
• run agent flows that call tools and external APIs safely,
• generate evaluation datasets and run batch evals,
• produce and manage fine-tuning datasets and jobs.

The server is designed to be privacy-first (runs where you deploy it), extensible (pluggable model and search backends), and integrates with Kiln’s open-source client libraries and desktop apps.

Features

• RAG pipelines: ingest documents, build vector indexes, and serve retrieval contexts to models.
• Agent orchestration: multi-step workflows, tool invocation, and safe tool authorization.
• Tool-calling interface: register callable tools (HTTP, SQL, shell, custom) and expose them to models.
• Evaluation framework: run automated model evaluations and track metrics across runs.
• Synthetic data generation: create labeled datasets for evals or fine-tuning via model programs.
• Fine-tuning management: prepare datasets, run fine-tune jobs, and track artifacts.
• Connectors: OpenAI, Ollama, local model runtimes, S3, FAISS/Annoy/Weaviate, and more.
• Team collaboration primitives: dataset/version control, issues, and sample reviews (via Kiln UI).
• REST + WebSocket APIs and a developer-friendly Python/JS SDK.

Installation / Configuration

Minimal steps to get a local development instance.

Clone the repository:

git clone https://github.com/Kiln-AI/Kiln.git
cd Kiln

Start with Docker Compose (recommended for local testing):

# from repo root
docker compose up --build

Environment variables (example .env):

# Model providers
OPENAI_API_KEY=sk-...
OLLAMA_SOCKET=/var/run/ollama.sock

# Storage / index
VECTOR_STORE_TYPE=faiss
STORAGE_PATH=/data/kiln

# Server
KILN_PORT=8080
KILN_ENV=development

Example config (kiln.yml) — registers providers, tools, and index settings:

providers:
  openai:
    api_key: ${OPENAI_API_KEY}
  ollama:
    socket: ${OLLAMA_SOCKET}

vector_stores:
  default:
    type: faiss
    path: ${STORAGE_PATH}/indexes/default

tools:
  - name: sql_db
    type: postgres
    conn: "postgresql://user:pass@host:5432/db"
  - name: web_fetch
    type: http
    allowed_domains: ["example.com"]

Run the backend (example):

KILN_CONFIG=kiln.yml docker compose up
# or run binary if available:
./kiln-server --config kiln.yml

API examples

Create a RAG task (HTTP):

curl -X POST http://localhost:8080/api/tasks \
  -H "Content-Type: application/json" \
  -d '{
    "type": "rag_query",
    "query": "How does authentication work?",
    "index": "default",
    "model": "openai:gpt-4o"
  }'

Call an agent that uses a tool:

curl -X POST http://localhost:8080/api/agents/run \
  -H "Content-Type: application/json" \
  -d '{
    "agent_id": "support-bot",
    "input": "Get the 30-day active users from analytics",
    "allow_tools": ["sql_db"]
  }'

Python example (requests):

import requests
resp = requests.post("http://localhost:8080/api/tasks", json={
    "type": "eval_run",
    "spec": "specs/sentiment_eval.yaml",
    "model": "openai:gpt-4o"
})
print(resp.json())

Available Tools / Resources

Built-in integrations and resources you can expect to use:

• Model providers: OpenAI, Ollama, local runtimes, OpenRouter-compatible endpoints.
• Vector stores: FAISS, Annoy, Weaviate, and cloud-hosted indexes.
• Tool types: HTTP fetch, SQL, shell (sandboxed), custom webhooks.
• Evaluation & specs: Evals runner, spec format for automated scoring.
• Fine-tuning helpers: dataset generators, formatting validators, job submission.
• SDKs: Python and JavaScript clients (MIT licensed).
• Docs & examples: in-repo examples for RAG, agents, evals, and fine-tuning.

Use Cases

• Documentation-assisted support bot (RAG)

  • Ingest product docs, index them, and serve retrieved passages as context to your model. Use agents to perform follow-up tool calls (create ticket, fetch user details).

• Agentic automation for analytics

  • Define an agent that can call SQL and an HTTP dashboard API. Ask a natural-language question and have the agent generate queries, run them, and summarize results.

• Iterative model evaluation and optimization

  • Generate synthetic test cases, run batch evals across multiple models/fine-tuned variants, and use optimizer tooling to find the best prompt+model tradeoff.

• Zero-code fine-tuning pipeline

  • Use the synthetic data tools to produce labeled examples, validate formats, submit fine-tune jobs, and automatically deploy artifacts to a local model runtime.

• Secure tool-calling in production

  • Register tools with explicit allowlists, audit calls in the server logs, and constrain outputs before returning them to users.

Component Summary