Overview

The Dagster MCP (Model Context Protocol) server provides a small, language-agnostic service that hosts pipeline execution contexts and exposes them over a stable network protocol. By running an MCP server alongside Dagster pipelines, you separate the orchestration/control plane from the execution/runtime plane. This makes it easier to test pipelines locally, run remote workers, and integrate Dagster into CI/CD and container orchestration environments.

The server acts as a single-process environment that can load code, provide metadata, and execute units of work (ops/solids) upon request from an orchestrator or test harness. Using an MCP server helps reduce variability between development and production runs, simplifies debugging by providing deterministic endpoints for introspection, and enables deployment patterns where pipelines run in isolated, reproducible runtimes.

Features

Lightweight process to host Dagster pipeline contexts and execute units of work
Network protocol (MCP) for remote execution and control
CLI and module entrypoints for start/stop and configuration
Configurable logging, port and address settings
Designed for local development, CI testing, and containerized deployments
Health checks and basic introspection endpoints for orchestration tools

Installation / Configuration

Install from PyPI (if published) or directly from the Dagster repository.

From PyPI (if available):

pip install dagster-dg-cli

Install from the repository (recommended for latest or development builds):

git clone https://github.com/dagster-io/dagster.git
cd dagster/python_modules/libraries/dagster-dg-cli
pip install -e .

Start the MCP server (example CLI usage — adjust for your installation):

# Using the installed CLI (example)
dagster-dg-cli mcp-server start --host 0.0.0.0 --port 50051 --workspace /path/to/repo

# Or via python module entrypoint
python -m dagster_dg_cli.mcp_server --host 0.0.0.0 --port 50051 --workspace /path/to/repo

Basic configuration options

# mcp-server.yaml (example)
host: 0.0.0.0
port: 50051
workspace: /path/to/your/dagster/repository
log_level: INFO
auth_token: "optional-static-token"   # optional, for simple auth
max_workers: 4

Environment variables (examples)

export MCP_HOST=0.0.0.0
export MCP_PORT=50051
export MCP_WORKSPACE=/workspace/repo
export MCP_LOG_LEVEL=DEBUG

Docker Compose example

version: "3.8"
services:
  mcp-server:
    image: python:3.10-slim
    volumes:
      - ./your-repo:/workspace/repo
    working_dir: /workspace/repo
    command: >
      bash -c "pip install -e python_modules/libraries/dagster-dg-cli &&
               python -m dagster_dg_cli.mcp_server --host 0.0.0.0 --port 50051 --workspace /workspace/repo"
    ports:
      - "50051:50051"
    environment:
      - MCP_LOG_LEVEL=INFO

Available Tools / Resources

CLI commands for starting and stopping the MCP server
Health check endpoint (HTTP or gRPC health service) for orchestration integration
Logging and debug output configurable by log level
Workspace loading: point the server at a Dagster repository or python package to expose defined pipelines
Token-based or environment-backed authentication for simple access control
Integration samples in Dagster repository for local dev and containerized setups

Tip: Check the repository’s examples directory for runnable samples and a minimal pipeline to exercise the server.

Use Cases

Local development and debugging
- Run an MCP server locally while iterating on ops and pipeline definitions. Your Dagster UI or test harness connects to a deterministic endpoint, making it easy to reproduce runtime issues.
- Example: Start the server on port 50051 and run unit tests that exercise remote execution against that server.
CI / integration testing
- Spin up an MCP server in CI to run full pipeline execution tests against a reproducible environment without needing the full scheduler/orchestrator stack.
- Example: In a GitHub Actions job, start the server as a background service, run integration tests that call pipelines through the MCP endpoint, then tear down the service.
Remote workers and scaling
- Use the MCP server to host execution environments for workers that perform compute-heavy tasks. The orchestration layer can dispatch work to one or many MCP-hosted runtimes.
- Example: Deploy MCP server containers in a Kubernetes StatefulSet or Deployment, each exposing an MCP endpoint for the orchestrator.
Controlled production deployment
- Deploy the MCP server as part of a production stack to ensure pipeline runs execute in an approved runtime image with pinned dependencies. This helps maintain consistency across dev, staging, and prod.

Quick troubleshooting

Server won’t start: Check logs for port binding errors and verify workspace path exists and contains a valid Dagster repository.
Clients can’t connect: Ensure network/firewall rules allow the configured host/port and confirm any auth token is set correctly on client and server.
Pipeline errors at runtime: Start the server in DEBUG logging to capture detailed stack traces and module-loading information.

Dagster MCP Server for Data Pipelines