Overview

This MCP (Model Context Protocol) server exposes desktop control primitives — screen capture, mouse movement/clicks and keyboard input — so external agents (models, automation scripts, or remote services) can observe and interact with a graphical desktop. It implements a small, HTTP-accessible API that maps MCP-style tool calls to local OS actions, enabling programmatic workflows that require visual context and direct input.

Use cases include automated GUI testing, remote assistance, robotic process automation (RPA), and integrating a multimodal model with a live desktop. By providing screenshots plus precise input commands, the server lets higher‑level controllers make decisions based on what’s visible and then reliably perform UI actions.

Features

Screen capture (full-screen or region) with configurable quality and format
Mouse control: move, click, drag, scroll, absolute and relative coordinates
Keyboard control: type text, press/release keys, modifier combos
Simple HTTP API compatible with MCP-style tool invocation
Optional authentication (API key)
Docker-friendly and headless-capable for server deployments
Configurable capture rate, image format and scaling

Installation / Configuration

Clone the repository and run the server locally, or use Docker.

Quick start (git + Python example):

# clone
git clone https://github.com/domdomegg/computer-use-mcp.git
cd computer-use-mcp

# create virtualenv (optional)
python3 -m venv .venv
source .venv/bin/activate

# install
pip install -r requirements.txt

# run (default port 8080)
python -m mcp_server --port 8080 --api-key YOUR_API_KEY

Docker:

# build
docker build -t mcp-server .

# run (expose port 8080, set API key)
docker run -p 8080:8080 -e MCP_API_KEY=YOUR_API_KEY --device /dev/uinput mcp-server

Environment variables / CLI options (typical):

MCP_API_KEY or –api-key: shared secret to authenticate requests
–port: port to listen on (default 8080)
–capture-format: png | jpeg (default png)
–capture-quality: integer 1–100 for JPEG
–capture-scale: scale factor for screenshots (reduce to save bandwidth)
–headless: allow running without interactive desktop (platform dependent)

Security note: Run behind an internal network or an authenticated proxy. Grant only trusted systems the API key because the server can fully control the host.

Available Tools

The server exposes a small set of tools (HTTP endpoints) that map to desktop actions. Below is a concise reference.

Tool / Endpoint	Purpose	Key Parameters
POST /screenshot	Capture the screen or a region	format, quality, x, y, width, height, scale
POST /mouse/move	Move cursor	x, y (absolute) or dx, dy (relative), duration
POST /mouse/click	Click mouse button	button: left
POST /mouse/drag	Click-and-drag	x1, y1, x2, y2, duration
POST /mouse/scroll	Scroll wheel	dx, dy
POST /keyboard/type	Type text	text, interval
POST /keyboard/press	Press/release key	key, action: press
GET /health	Server health and capabilities	—

Example curl: capture a JPEG screenshot

curl -sS -H "Authorization: Bearer YOUR_API_KEY" \
  -X POST "http://localhost:8080/screenshot" \
  -H "Content-Type: application/json" \
  -d '{"format":"jpeg","quality":70,"scale":0.5}' \
  --output screen.jpg

Example curl: move mouse and click

curl -H "Authorization: Bearer YOUR_API_KEY" -H "Content-Type: application/json" \
  -X POST http://localhost:8080/mouse/move -d '{"x":400,"y":300,"duration":0.2}'

curl -H "Authorization: Bearer YOUR_API_KEY" -H "Content-Type: application/json" \
  -X POST http://localhost:8080/mouse/click -d '{"button":"left","clicks":1}'

Python client example:

import requests

API = "http://localhost:8080"
HEADERS = {"Authorization": "Bearer YOUR_API_KEY", "Content-Type": "application/json"}

# take a screenshot
r = requests.post(f"{API}/screenshot", headers=HEADERS, json={"format":"png"})
with open("screen.png", "wb") as f:
    f.write(r.content)

# type text
requests.post(f"{API}/keyboard/type", headers=HEADERS, json={"text":"Hello, world!", "interval":0.05})

Use Cases

Automated GUI test flows: capture the app UI, verify visual state, and exercise keyboard/mouse sequences to validate behavior.
Model-driven agents: feed screenshots to a vision-enabled model, have the model decide the next action, and then execute it via mouse/keyboard tools.
RPA for legacy apps: drive applications that lack APIs by scripting interactions with visible UI elements.
Remote assistance & demos: allow a trusted remote agent to capture and interact with a desktop for troubleshooting or demonstrations.
Scheduled unattended tasks: run workflows that require UI steps (e.g., exporting from a GUI-only tool) on a headless machine with an X/virtual framebuffer.

Tips and Best Practices

Reduce screenshot resolution (scale) and use JPEG for high-frame-rate workflows to save bandwidth.
Use explicit coordinates or image-matching at the automation layer to avoid brittle timing-based sequences.
Combine screenshot captures with a vision library (OpenCV) to locate controls, then call mouse/keyboard endpoints with the computed coordinates.
Lock down network access and rotate API keys in production; consider running behind a VPN.

For the latest source, issues, and contribution guidelines, see the project repository: https://github.com/domdomegg/computer-use-mcp

MCP Server for Screen, Mouse, Keyboard Control