Databricks MCP Server: LLM SQL Queries & Jobs

Overview

The Databricks MCP Server exposes a small, compatible Model Context Protocol (MCP) endpoint that lets language models and agent frameworks interact with a Databricks workspace. It provides two primary capabilities: running SQL queries against Databricks SQL endpoints and listing or inspecting Databricks job executions. By translating tool calls into Databricks REST API requests, the server makes it straightforward to augment LLM-driven applications with live data access and job orchestration visibility.

This server is useful when building AI agents that must fetch data, validate results, or kick off / inspect CI-style workloads inside Databricks. Instead of hard-coding Databricks API calls into your application logic, you can register the MCP server as a tool provider for an LLM runtime. That keeps security boundaries, centralizes auditing, and simplifies the LLM-to-Databricks integration.

Features

• Run ad-hoc SQL queries against Databricks SQL warehouses (execute and return results).
• List Databricks jobs and recent job runs.
• Fetch detailed information for a specific job run (status, timestamps, results metadata).
• Simple MCP-compatible tool manifest for LLM tool discovery.
• Configurable via environment variables and compatible with container deployments.
• Lightweight: intended for running alongside an LLM agent or gateway.

Installation / Configuration

Prerequisites: Python 3.9+, Databricks personal access token with appropriate permissions.

Clone and install:

git clone https://github.com/JordiNeil/mcp-databricks-server.git
cd mcp-databricks-server
python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

Environment variables

Example .env:

DATABRICKS_HOST=my-workspace.cloud.databricks.com
DATABRICKS_TOKEN=dapiXXXXXXXXXXXXXXXX
MCP_PORT=8080

Run the server (development):

export DATABRICKS_HOST=...
export DATABRICKS_TOKEN=...
uvicorn mcp_server.main:app --host 0.0.0.0 --port ${MCP_PORT:-8080}

Docker

docker build -t mcp-databricks .
docker run -e DATABRICKS_HOST=my-workspace.cloud.databricks.com \
           -e DATABRICKS_TOKEN=dapiXXXX \
           -p 8080:8080 \
           mcp-databricks

Security note: keep the Databricks token secret. Consider using a secrets manager or injecting tokens at runtime instead of baking them into images.

Available Tools

The MCP server exposes a small set of tools intended for common LLM interactions with Databricks. Each tool includes a short description and a JSON schema for its arguments (used by LLM tool-selection).

• databricks.query_sql
  • Description: Execute a SQL string on a Databricks SQL warehouse and return tabular results (rows, columns).
  • Parameters: { sql: string, limit?: integer }
• databricks.list_jobs
  • Description: Return a list of jobs configured in the workspace; supports pagination.
  • Parameters: { page_size?: integer, page_token?: string }
• databricks.get_job_run
  • Description: Fetch detailed information about a particular job run (state, timings, result).
  • Parameters: { run_id: string }

Example tool manifest (JSON):

{
  "tools": [
    {
      "name": "databricks.query_sql",
      "description": "Run SQL on Databricks and return results",
      "parameters": {
        "type": "object",
        "properties": {
          "sql": { "type": "string" },
          "limit": { "type": "integer" }
        },
        "required": ["sql"]
      }
    },
    {
      "name": "databricks.list_jobs",
      "description": "List Databricks jobs",
      "parameters": { "type": "object" }
    },
    {
      "name": "databricks.get_job_run",
      "description": "Get a job run's details",
      "parameters": { "type": "object", "properties": { "run_id": { "type": "string" } }, "required": ["run_id"] }
    }
  ]
}

Internally each tool maps to the corresponding Databricks REST endpoint and normalizes responses to JSON consumable by LLMs.

Use Cases

1. Interactive analytics via an LLM

   • Prompt: “How many active users did we have last week and show the top three countries?”
   • LLM selects databricks.query_sql with SQL: SELECT country, COUNT(*) AS users FROM events WHERE event_date BETWEEN … GROUP BY country ORDER BY users DESC LIMIT 3

2. Automated incident triage

   • An agent checks recent Databricks job runs and surfaces failing jobs.
   • Calls databricks.list_jobs, then for each candidate uses databricks.get_job_run to fetch the latest run and its state.

3. Notebookless data checks in pipelines

   • A CI workflow uses the server to run quick validation queries before promoting schemas or data. The server returns structured results the agent can reason over.

4. Human-in-the-loop exploration

   • Analysts ask an LLM to “show me anomalies in yesterday’s ETL runs.” The agent lists jobs and inspects run outcomes to compile a human-friendly summary.

Example curl call (execute SQL):

curl -X POST http://localhost:8080/tool/execute \
  -H "Content-Type: application