Computer Control MCP Server — Mouse, Keyboard, OCR

Overview

This project implements a Model Context Protocol (MCP) server that exposes basic computer-control primitives—mouse, keyboard and OCR—so an MCP-capable agent or client can control a desktop environment programmatically. It wraps common desktop automation and OCR libraries to provide a small set of tools that are callable over a simple HTTP/MCP-style API. The server is intended for local automation, testing, accessibility tooling and agent-driven workflows where an AI or automation system needs to interact with a graphical desktop.

The implementation uses well-known building blocks (PyAutoGUI for input automation, RapidOCR and ONNX Runtime for local OCR) while keeping the surface area minimal so you can run it on a workstation without heavy external services. Because it exposes discrete tool endpoints, it can be plugged into an MCP-enabled agent pipeline or called directly from scripts and CI jobs that require GUI-level interactions and on-screen text extraction.

Features

• Programmatic mouse control: move, click, drag, scroll, position queries
• Keyboard interaction: type text, key presses, hotkeys
• On-screen OCR: extract text from full-screen or region screenshots using RapidOCR and ONNX Runtime
• Simple JSON-based HTTP/MCP interface for tool invocation
• Minimal runtime dependencies and local execution (no cloud OCR required)
• Example clients and payloads for curl and Python requests

Installation / Configuration

Prerequisites: Python 3.8+, pip, and a desktop environment where automated input is permitted.

Quick start:

git clone https://github.com/AB498/computer-control-mcp.git
cd computer-control-mcp
python -m venv .venv
source .venv/bin/activate   # Windows: .venv\Scripts\activate
pip install -r requirements.txt

Run the server (examples — the actual entrypoint may vary):

# If the repo provides a simple script
python server.py

# Or using an ASGI server if the app exports an app object
uvicorn server:app --host 0.0.0.0 --port 8080

Configuration options (examples; actual keys may differ by release):

• OCR_MODEL_PATH: path to a local ONNX OCR model (optional if bundled)
• HOST / PORT: network binding for the HTTP/MCP server
• ENABLE_GUI: whether GUI automation is enabled (sensible default: true on local machines)

Set environment variables before starting:

export HOST=127.0.0.1
export PORT=8080
export OCR_MODEL_PATH=/path/to/ocr.onnx
python server.py

To install optional GPU-accelerated ONNX Runtime:

pip install onnxruntime-gpu

Available Tools

The server exposes three primary tools via HTTP/MCP endpoints. Example endpoints and minimal payloads are shown here; consult the repository for exact routes and schema.

Table: tools overview

Example: move mouse to coordinates with curl

curl -X POST http://127.0.0.1:8080/tools/mouse \
  -H "Content-Type: application/json" \
  -d '{"action":"move","x":400,"y":300,"duration":0.2}'

Example: type a string with Python requests

import requests
resp = requests.post("http://127.0.0.1:8080/tools/keyboard",
                     json={"action":"type","text":"Hello, world!"})
print(resp.json())

Example: OCR a screen region

curl -X POST http://127.0.0.1:8080/tools/ocr \
  -H "Content-Type: application/json" \
  -d '{"region": {"left":100,"top":200,"width":400,"height":100}, "language":"en"}'

Typical request fields

• action: operation to run (move, click, type, capture, etc.)
• x, y, duration: mouse coordinates and movement time
• text: string to type
• region: bounding box for OCR capture
• options: additional flags (confidence thresholds, image preprocessing)

Use Cases

• Automated UI testing: script complex user interactions (drag/drop, hotkeys) and validate on-screen text via OCR.
• Assistive applications: let an agent or automation layer operate the desktop on behalf of a user with limited mobility.
• Agent-driven workflows: AI agents that need to inspect application state visually and interact with UI elements can call the MCP tools to read and control the screen.
• Data extraction from applications: capture regions of proprietary UIs and run offline OCR to harvest text for logging or reporting.
• Remote demonstration and debugging: reproduce sequences of input programmatically and capture the visual output for diagnostics.

Security and safety notes

• This server can control your mouse/keyboard and read your screen—run it only in trusted environments.
• Consider binding to localhost and using access controls if exposing the server on a network.

For implementation details,