Databricks Smart SQL MCP Server for Unity Catalog

Overview

This MCP (Model Context Protocol) server integrates with Databricks Unity Catalog to serve catalog metadata to LLMs and agent frameworks. By exposing table schemas, column types, sample rows and basic statistics through a lightweight HTTP API, the server enables models to generate smarter, safer, and more efficient SQL for ad-hoc queries and data exploration without requiring direct database credentials in prompts.

The project is designed for developers building LLM-powered data tools or agents that need context about available datasets. Instead of asking models to guess table names or column types, the MCP server provides structured, up-to-date metadata so models can produce valid SQL, avoid full-table scans, and prioritize relevant columns and filters.

Features

• Exposes Unity Catalog metadata (catalogs, schemas, tables, columns)
• Returns sample rows and simple column statistics to guide query planning
• Search capability for tables and columns by name and description
• LLM-friendly context formatting (compact schema summaries)
• Cache support to reduce repeated catalog calls
• Authentication via Databricks personal access token (PAT)
• Easy to self-host (Python + Docker examples included)

Installation / Configuration

Prerequisites:

• Databricks workspace with Unity Catalog enabled
• Databricks personal access token (PAT) with catalog read permissions
• Python 3.9+ (or Docker)

Clone and install locally:

git clone https://github.com/RafaelCartenet/mcp-databricks-server.git
cd mcp-databricks-server
python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

Environment variables (example):

export DATABRICKS_HOST="https://adb-123456789012345.18.azuredatabricks.net"
export DATABRICKS_TOKEN="dapiXXXXXXXXXXXXXXXXXXXXXXXX"
export DEFAULT_CATALOG="hive_metastore"   # optional: default catalog or schema
export PORT=8080

Run with Uvicorn (development):

uvicorn app.main:app --host 0.0.0.0 --port ${PORT:-8080} --reload

Run with Docker:

docker build -t mcp-databricks .
docker run -e DATABRICKS_HOST="$DATABRICKS_HOST" \
           -e DATABRICKS_TOKEN="$DATABRICKS_TOKEN" \
           -p 8080:8080 mcp-databricks

Configuration options typically include: host, token, default catalog/schema, cache TTL, and optional rate limiting. Check the repository README or config file in the project for exact environment keys.

Available Resources

• GitHub repository: https://github.com/RafaelCartenet/mcp-databricks-server
• Databricks Unity Catalog docs: https://docs.databricks.com/data-governance/unity-catalog/index.html
• Model Context Protocol (MCP) spec: implementers should consult the MCP spec used by their agent framework or LLM provider

Typical HTTP resources exposed by the server (names may vary by implementation):

Use Cases

1. LLM-assisted ad-hoc SQL generation

• Problem: Users ask an LLM to write SQL but the model guesses table or column names, producing invalid queries.
• Solution: The agent queries the MCP server for table and column metadata, then asks the model to generate SQL constrained to those names and types. The result is executable SQL with fewer errors and fewer unnecessary scans.

2. Data discovery and exploration

• Problem: Analysts don’t know which tables contain relevant fields for a task.
• Solution: Use the server’s search endpoint to find tables containing specific columns or keywords, fetch sample rows, and inspect column statistics before running larger queries.

3. Cost-conscious query planning

• Problem: Blind queries can scan large tables and incur cost.
• Solution: Inspect column statistics and sample previews to design selective WHERE clauses or suggest partitions, thereby reducing processed data.

4. Building LLM agents and apps

• Problem: Agents need structured context to safely interact with databases.
• Solution: Integrate the MCP server as a tool in your agent runtime so that the LLM gets machine-readable metadata and can reason about available datasets.

Quick Example (curl)

Fetch table columns:

curl -s -H "Authorization: Bearer $DATABRICKS_TOKEN" \
  "http://localhost:8080/tables/my_catalog.my_schema.my_table/columns"

Sample JSON response (illustrative):

{
  "table": "my_catalog.my_schema.my_table",
  "columns": [
    {"name": "id", "type": "BIGINT"},
    {"name": "created_at", "type": "TIMESTAMP"},
    {"name": "amount", "type": "DECIMAL(10,2)"}
  ],
  "row_count_estimate": 123456
}

Next Steps

• Review the repo for detailed configuration options and any supported authentication flows.
• Integrate the MCP endpoints into your LLM prompt-engineering or agent tooling to provide accurate context before SQL generation.
• Add caching and access controls as needed for production usage.