Browser MCP Server for Puppeteer Automation

Overview

This Browser MCP (Model Context Protocol) server provides a lightweight bridge between model-driven agents and a real Chromium browser instance using Puppeteer accessibility data. It exposes a small set of tools over MCP so models can inspect the accessibility tree, interact with DOM elements, and optionally use a vision-enabled mode (pixel screenshots + OCR/segmentation) when accessibility signals are insufficient.

The server is designed to be fast, cross-platform, and easy to integrate into model workflows for web automation, scraping, RPA, or accessibility testing. It can run headless or headful, accepts flexible configuration (browser executable, viewport, user agent, etc.), and focuses on providing stable semantic DOM information rather than brittle visual scraping only.

Features

• Puppeteer-based Chromium control with accessible DOM snapshots (A11y tree)
• Optional vision mode: screenshots plus basic OCR / segmentation hooks
• Cross-platform configuration (custom browser path, headless/headful, env)
• Small, focused set of MCP tools for:
  • accessibility snapshots
  • DOM queries and interactions (click, type, focus)
  • screenshots and visual context
• Designed to be used as an MCP server for model-driven agents (text/vision models)
• Lightweight and fast: single-process server with minimal dependencies

Installation / Configuration

Clone the repository and install dependencies:

git clone https://github.com/bytedance/UI-TARS-desktop.git
cd UI-TARS-desktop/packages/agent-infra/mcp-servers/browser
npm install

Start the server with a local configuration file:

# start with node
node ./src/server.js --config ./config.json

# or using npm script if provided
npm run start -- --config ./config.json

Example configuration (config.json):

{
  "port": 3000,
  "headless": true,
  "defaultViewport": { "width": 1280, "height": 800 },
  "browserExecutablePath": "/path/to/chrome",
  "enableVision": false,
  "visionOptions": {
    "ocr": true,
    "segmentation": false
  },
  "mcp": {
    "transport": "websocket",
    "authToken": "replace-me"
  }
}

Quick Docker example:

Dockerfile (minimal)

FROM node:18-alpine
WORKDIR /app
COPY . .
RUN npm ci --production
CMD ["node", "./src/server.js", "--config", "/app/config.json"]

Build & run:

docker build -t mcp-browser .
docker run -p 3000:3000 -v /local/config.json:/app/config.json mcp-browser

Configuration reference (common fields):

Available Tools / Resources

The server exposes a small, purpose-driven toolset via MCP. Tool names below are illustrative — check the server’s actual tool registry endpoint for exact names.

• getAccessibilitySnapshot
  • Returns the Puppeteer accessibility tree for the current page.
• querySelector / querySelectorAll
  • Locate nodes in the accessibility/dom tree using selectors / predicates.
• click / focus / type
  • Simulate user interactions on resolved elements.
• screenshot
  • Full or region screenshot; useful when enableVision is true.
• getDOM
  • Return structured DOM fragments (attributes, roles, text).
• startVision / stopVision
  • Toggle vision processing: take screenshots and attach OCR/segmentation metadata.

Typical response format: JSON object containing tool result and metadata (node ids, bounding boxes, text).

Use Cases

• Web scraping with semantic reliability
  • Use getAccessibilitySnapshot to extract product titles, prices, and ARIA labels instead of brittle CSS selectors. Combine with querySelector to refine selection.
• Form autofill and RPA
  • Locate inputs by role/label via the accessibility tree and use type/click to complete forms in headful or headless mode.
• Accessibility audits
  • Run periodic snapshots across pages and surface missing roles, labels, or focusable elements.
• Vision-assisted interactions
  • For applications with dynamic canvas content or non-semantic markup, enable vision mode to combine screenshots + OCR and then map visual text back to nearest DOM nodes.
• Model-driven agents
  • Connect an LLM or multimodal model via MCP transport (WebSocket/HTTP) to ask high-level instructions like “log in and save the first 3 product links” while the server translates those into accessibility queries and interactions.

Quick example (conceptual)

An agent could invoke the following MCP request to get product titles:

Request:

{
  "type": "tool.invoke",
  "tool": "getAccessibilitySnapshot",
  "args": { "rootUrl": "https://example.com/category" }
}

Then use:

• querySelectorAll to find nodes with role=link and collect text
• screenshot (if needed) for visual verification

Notes:

• This server is intended to be integrated as a backend component in model-driven systems. Verify the available tool names and transport details by inspecting the server’s runtime registry or the /tools endpoint after startup.