Overview

The VisionAgent MCP Server is a lightweight service that exposes multimodal image, video, and document reasoning capabilities to large language models (LLMs) using the Model Context Protocol (MCP). Instead of embedding raw pixels or whole documents into the LLM prompt, the server provides a set of deterministic, queryable tools — e.g., OCR, object detection, segmentation, keyframe extraction, layout parsing — that LLM agents can call to obtain structured observations and visual context. This makes it simpler to build chains where text and visual reasoning are tightly integrated.

For developers, the MCP server acts as the bridge between your media assets and an LLM-based agent. It accepts MCP-compliant tool requests over HTTP, runs the requested visual or document operations (locally or via configured backends), and returns results in structured JSON. This pattern reduces prompt engineering complexity, keeps heavy media processing out of the LLM context, and enables reproducible multimodal workflows.

Features

Exposes multimodal operations as MCP tools (image, video, document)
OCR and text extraction for scanned pages, PDFs, and images
Object detection and bounding-box queries with attribute metadata
Semantic segmentation and mask outputs for pixel-level reasoning
Video tools: keyframe extraction, frame-wise detection, temporal queries
Document layout parsing: pages, blocks, tables, and semantic regions
Embeddings for images and document pages for similarity search
HTTP API with MCP-compliant request/response structure
Configurable model/backends and optional GPU acceleration
Docker and local development options for easy deployment

Installation / Configuration

Clone the repository and install dependencies, or run with Docker.

Local (Python/pip)

git clone https://github.com/landing-ai/vision-agent-mcp.git
cd vision-agent-mcp
python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
# configure env vars or config file (see next section)
uvicorn mcp_server.main:app --reload --host 0.0.0.0 --port 8080

Docker

# build
docker build -t vision-agent-mcp .
# run (exposes port 8080)
docker run -e CONFIG_PATH=/app/config.yaml -p 8080:8080 vision-agent-mcp

Configuration (env or YAML)

# config.yaml (example)
server:
  host: 0.0.0.0
  port: 8080

backends:
  ocr:
    type: tesseract
    path: /usr/bin/tesseract
  detector:
    type: yolov5
    model: yolov5s.pt
  segmentation:
    type: detectron2
    model: mask_rcnn

auth:
  api_keys:
    - sk-...

Provide paths to model weights or configure remote backends (e.g., hosted vision APIs).
Optionally configure API keys for request authentication.
GPU-enabled backends require CUDA-capable images and drivers when using Docker.

Available Tools / Resources

The server exposes tools as MCP actions. Common available tools include:

image.ocr: Extract text and bounding boxes from an image.
image.caption: Return a short caption describing an image.
image.detect_objects: Return detected objects and bbox metadata.
image.segment: Return segmentation masks or polygons for regions.
image.embed: Produce an embedding vector for image similarity.
video.keyframes: Extract representative frames from a video.
video.track_object: Locate and track an object across frames.
document.parse_layout: Extract pages, blocks, tables, and reading order.
document.extract_table: Parse tables into CSV/structured rows.

Example HTTP endpoints

# MCP request to call a tool
curl -X POST 'http://localhost:8080/mcp/execute' \
  -H 'Authorization: Bearer sk-...' \
  -H 'Content-Type: application/json' \
  -d '{"tool":"image.ocr","input":{"image_url":"https://.../invoice.jpg"}}'

Responses follow MCP-style JSON with standardized fields: tool output, metadata, diagnostics, and optional attachments (e.g., mask as PNG).

Use Cases

Visual question answering: An LLM asks targeted questions about an image (e.g., “How many chairs are in the room?”) and calls image.detect_objects to get counts and attributes, then composes the answer.
Document understanding: Extract structured data from invoices or forms by combining document.parse_layout and image.ocr to obtain key-value pairs, tables, and normalized fields for downstream workflows.
Video summarization: Use video.keyframes to collect representative frames, then run image.caption or image.detect_objects to generate scene-level summaries or indexable metadata.
Assisted annotation: Preprocess images with object detection and segmentation tools to accelerate manual labeling and produce consistent initial annotations.
Retrieval-augmented multimodal search: Index image/page embeddings from image.embed/document.page_embed and perform nearest-neighbor lookups to retrieve visually similar content for an LLM to reason over.

Getting Help & Contributing

Repository: https://github.com/landing-ai/vision-agent-mcp
Open issues and pull requests on GitHub for bugs, feature requests, or integration questions.
Read the repo’s examples and tests directory for sample MCP requests and agent integration patterns.

This MCP server is intended as an extensible foundation for multimodal LLM agents. Start by configuring the backends you need, then expose the tools your agents will call to keep visual reasoning structured, auditable, and reproducible.

VisionAgent MCP Server for Image, Video, Document Reasoning