Overview

This MCP server exposes HuggingFace vision models—primarily zero-shot object detection—behind a small HTTP service that implements the Model Context Protocol (MCP). The server makes it straightforward to bring vision capabilities into LLM-driven workflows and toolchains by wrapping popular HuggingFace models into a consistent, tool-like interface that LLM tool runners and orchestrators can call.

You can run the server locally or in a container, point it at models hosted on the HuggingFace Hub (or local checkpoints), and call it from agents, pipelines, or direct HTTP clients. The server accepts images (URLs or base64) and zero-shot label sets, returning detection results (bounding boxes, labels, and confidence scores) in a structured JSON format compatible with MCP-based tool integrations.

Features

Zero-shot object detection via HuggingFace models (no training required)
MCP-compatible API for easy integration with LLM tool runners and orchestration layers
Supports image inputs via URL or base64 payload
Configurable model selection (HuggingFace Hub model name or local path)
Docker and local Python/uvicorn deployment options
Options for device selection (cpu / cuda) and model preload / caching
JSON responses with detections (label, score, bounding box) suitable for downstream reasoning

Installation / Configuration

Clone and install with pip:

git clone https://github.com/groundlight/mcp-vision.git
cd mcp-vision
python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

Run with uvicorn (example):

export HUGGINGFACEHUB_API_TOKEN="hf_xxx"   # if using private Hub models
uvicorn mcp_vision.main:app --host 0.0.0.0 --port 8080

Docker (recommended for production / quick start):

docker run -p 8080:8080 \
  -e HUGGINGFACEHUB_API_TOKEN="hf_xxx" \
  ghcr.io/groundlight/mcp-vision:latest

Example configuration (config.yaml):

model_name: "open-mmlab/detr-resnet-50"
device: "cuda"               # or "cpu"
preload: true
port: 8080
allowed_origins:
  - "*"

Environment variables (common):

Variable	Purpose
HUGGINGFACEHUB_API_TOKEN	Token to access private HF Hub models (optional)
MCP_SERVER_PORT	Port to listen on (default: 8080)
MCP_MODEL_NAME	Default HuggingFace model name
MCP_DEVICE	“cpu” or “cuda”

Available Resources

The server exposes resources useful to LLMs and pipelines. Typical endpoints:

POST /mcp/invoke — MCP-compatible JSON request. Use this when integrating with an LLM tool runner expecting MCP semantics.
POST /detect — Simpler direct endpoint for detection use: accepts image and label candidates.
GET /health — Basic health and model load status.

Supported model capabilities (examples):

zero_shot_object_detection — detect objects given arbitrary label candidates
image_classification — top-k classification on input image
image_captioning — single-sentence captioning (if a captioning model is configured)

Responses follow JSON structures with a detections array containing objects like:

{
  "detections": [
    {
      "label": "person",
      "score": 0.98,
      "bbox": [x_min, y_min, x_max, y_max]
    }
  ]
}

Use Cases

Augment an LLM with visual context: call the MCP server from an agent when the LLM needs to reason about an image (e.g., “how many apples are in this photo?”). Provide zero-shot label candidates like [“apple”, “orange”, “banana”] to detect unlabeled items without retraining.
Content moderation and triage: quickly scan uploaded images for certain object categories (weapons, logos, restricted items) by supplying a list of target labels for zero-shot detection.
Retail / inventory checks: detect items on shelves by providing the SKU or product names as zero-shot labels, enabling lightweight visual search workflows without custom models.
Robotics / automation: supply camera frames to the MCP server and return structured detections that an agent can use for planning or navigation.

Example: Zero-shot detection (curl & Python)

Curl example (image URL + labels):

curl -X POST "http://localhost

MCP Server: HuggingFace Zero-Shot Object Detection