Kubeflow Spark Job Performance MCP Server

Overview

Kubeflow Spark Job Performance MCP Server is a lightweight service that ingests Apache Spark event logs and exposes structured performance context via the Model Context Protocol (MCP). It helps teams running Spark-based AI and data workflows to surface actionable performance diagnostics — for example, slow stages, skewed tasks, inefficient shuffle patterns, and resource bottlenecks — in a machine-readable format that can be consumed by ML tooling and orchestration systems.

By converting Spark history data into MCP-compatible context, the server enables automated tooling (dashboards, pipeline steps, or alerting services) to reason about job health and to recommend optimizations. This is useful for Kubeflow environments where reproducibility and observability of training and preprocessing jobs matter for reliable model delivery.

Features

• Parse and normalize Spark event logs from local disk or object storage (GCS, S3).
• Produce MCP-compliant context bundles that summarize job, stage, and task-level performance.
• REST API to fetch diagnostics, heatmaps, and suggested optimization points.
• Integrates with Kubeflow pipelines and other MCP consumers to feed automated decision steps.
• Configurable storage backends and authentication for cloud storage.
• Lightweight container suitable for Kubernetes deployments.

Installation / Configuration

Below are common ways to run the MCP server. Adjust environment variables and storage settings to match your environment.

Run with Docker (local or testing)

docker run -p 8080:8080 \
  -e STORAGE_BACKEND=gcs \
  -e GCS_BUCKET=my-spark-logs \
  -e MCP_BIND_ADDRESS=0.0.0.0:8080 \
  ghcr.io/kubeflow/mcp-apache-spark-history-server:latest

Minimal Kubernetes deployment (example)

apiVersion: apps/v1
kind: Deployment
metadata:
  name: spark-mcp-server
spec:
  replicas: 1
  selector:
    matchLabels:
      app: spark-mcp-server
  template:
    metadata:
      labels:
        app: spark-mcp-server
    spec:
      containers:
      - name: mcp-server
        image: ghcr.io/kubeflow/mcp-apache-spark-history-server:latest
        ports:
        - containerPort: 8080
        env:
        - name: STORAGE_BACKEND
          value: "gcs"
        - name: GCS_BUCKET
          value: "my-spark-logs"
        - name: MCP_BIND_ADDRESS
          value: "0.0.0.0:8080"

Helm (if provided by your environment)

helm repo add kubeflow https://kubeflow.github.io/manifests
helm install spark-mcp kubeflow/mcp-apache-spark-history-server \
  --set storage.backend=gcs \
  --set storage.gcs.bucket=my-spark-logs \
  --set service.port=8080

Configuration reference

API example (fetch performance summary)

curl "http://<host>:8080/api/v1/performance?jobId=application_1234567890"

Note: endpoint path is illustrative — check the live server’s /openapi or /docs for exact endpoints.

Available Resources

• Source code and issues: https://github.com/kubeflow/mcp-apache-spark-history-server
• Model Context Protocol (MCP) specification: refer to your MCP consumer docs or Kubeflow MCP docs for schema details
• Example Spark event logs: use Spark job history server or event log directory in your cluster
• Kubernetes manifests: include the deployment example above — adapt RBAC and secret mounts for cloud credentials

Use Cases

• Pinpointing slow stages in training jobs

  • Run the MCP server on your cluster logs, then query the MCP API for stage-level summaries. The server will highlight stages with high shuffle write/read or long task durations so you can target code or partitioning changes.

• Detecting data skew and task stragglers

  • Use the task duration distributions produced by the MCP server to identify outlier tasks. Combine that data with input partition statistics to re-balance partitioning or increase parallelism.

• Comparing repeated runs for regression detection

  • Persist MCP context outputs for two runs of the same pipeline to automatically detect performance regressions (e.g., increased GC time, longer shuffle times, or higher executor CPU usage).

• Automating pipeline decisions

  • Integrate the MCP server into Kubeflow Pipelines: after a training step, fetch the MCP summary and trigger a conditional step (e.g., notify, rollback parameters, or run a tuning job) when thresholds are exceeded.

• Cost optimization

  • Aggregate executor time and resource utilization from MCP summaries to identify over-provisioned jobs and recommend instance size reductions or executor configuration changes.

Tips

• Ensure Spark is configured to write detailed event logs by setting spark.eventLog.enabled and pointing spark.eventLog.dir to object storage or a volume accessible by the MCP server.
• When using cloud storage, mount or provide credentials via Kubernetes Secrets and set appropriate RBAC for access.
• Start with LOG_LEVEL=DEBUG only when troubleshooting — verbose logs can be large.

For development, consult the repository to run test suites and to see example event logs that exercise parsing and MCP export paths.