Puppeteer Vision MCP Server: Web-to-Markdown Automation

Overview

Puppeteer Vision MCP Server is a lightweight service that converts live webpages into clean, high-quality Markdown. It runs a Puppeteer-controlled headless browser to fully render pages (including JavaScript), interact with dynamic elements, and extract the semantic content you need for downstream processing. The server exposes this functionality via the Model Context Protocol (MCP), making it easy to call from LLM agents, automation pipelines, or custom tooling.

Because it renders pages in a real browser context, Puppeteer Vision can handle cookies, interactive UI flows (clicks, form fills), and client-side content that static scrapers miss. It also provides mechanisms to surface or work around captchas: by default it will detect blocking prompts and either pause for manual resolution or forward them to configurable solving services. This makes it particularly useful for ingestion into knowledge bases, content auditing, dataset generation, or as a tool for agents that need reliable webpage context in Markdown form.

Features

• Converts fully-rendered webpages to clean, readable Markdown
• Runs in a real Chromium session via Puppeteer (JS/CSS executed)
• Handles cookies and persistent sessions for authenticated scraping
• Interacts with pages: click buttons, fill forms, follow paginated links
• Captcha detection with optional integration hooks for external solvers or manual resolution workflows
• Exposes functionality over MCP-compatible HTTP endpoints for easy agent integration
• Configurable timeouts, concurrency, and browser launch arguments
• Optional screenshot capture and raw HTML export alongside Markdown
• Docker-ready and easy to run locally or in server environments

Installation / Configuration

Requirements: Node.js (16+), optional Docker, and enough memory for headless Chromium.

Clone and install:

git clone https://github.com/djannot/puppeteer-vision-mcp.git
cd puppeteer-vision-mcp
npm install

Environment variables (example .env):

PORT=3000
HEADLESS=true          # true or false
CONCURRENCY=2
TIMEOUT_MS=60000
COOKIE_STORE=./cookies.json
CHROME_EXECUTABLE_PATH=    # optional path to Chrome/Chromium
CAPTCHA_SOLVER_API_KEY=    # optional external solver key
LOG_LEVEL=info

Start the server:

# development
npm start

# with a specific env file
PORT=3000 HEADLESS=true npm start

Docker (simple example):

docker build -t puppeteer-vision-mcp .
docker run -p 3000:3000 \
  -e PORT=3000 \
  -e HEADLESS=true \
  -e TIMEOUT_MS=60000 \
  puppeteer-vision-mcp

Configuration options are typically provided via environment variables or a JSON config file. Key options include concurrency, timeout, headless mode, and captcha/solver settings.

Available Tools / Resources

Typical endpoints and tools exposed by the MCP server:

Common tool names and payload keys:

• convert_page_to_markdown: { url, waitForSelector?, actions?, cookies? }
• screenshot_page: { url, fullPage?, selector? }
• fetch_raw_html: { url, waitForNetworkIdle? }

Returned payloads typically include:

• markdown (string)
• html (string, optional)
• screenshot (base64 PNG, optional)
• logs / metadata (timings, navigation steps)

Use Cases

• Content ingestion for LLMs: Convert news articles, docs, or blogs to compact Markdown before feeding into vector stores or summarization pipelines.

  • Example: an agent invokes convert_page_to_markdown with url=https://example.com/article, receives cleaned Markdown, then inserts it into a knowledge base.

• Scraping behind login flows: Use persistent cookies and scripted interactions to authenticate and extract content from member-only pages.

  • Example actions: open login page, fill username/password selectors, click submit, then convert the resulting page.

• Archival and SEO audits: Capture both Markdown and screenshots for documentation or SEO analysis.

  • Example: POST to /tools/screenshot and /tools/convert for the same URL to get visual and textual snapshots.

• Dataset generation for ML: Generate standardized Markdown from thousands of pages for downstream NLP tasks (summarization, classification).

  • Use concurrency and timeout settings to control throughput and avoid overload.

• Agent tool for LLMs (MCP): Use the MCP interface to let language models request webpage context dynamically as part of a larger multi-tool workflow.

Examples

Simple curl call to convert a page:

curl -X POST http://localhost:3000/tools/convert \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com/article",
    "waitForSelector": "article",
    "actions": []
  }'

Node.js example using fetch:

const fetch = require('node-fetch');

async function convert(url) {
  const res = await fetch('http://localhost:3000/tools/convert', {
    method: 'POST',
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify({ url, waitForSelector: 'article' })
  });
  const result = await res.json();
  console.log(result.markdown);
}

convert('https://example.com/article');

Notes and Best Practices

• Captcha handling: the server can detect captchas and supports hooks for external solver services or manual resolution; for large-scale scraping you should configure an appropriate solver or human-in-the-loop flow.
• Respect robots.txt and site terms of service; control request rate with concurrency and delays.
• Persist cookies if you need long-lived sessions. Store cookie files securely.
• For production, place the server behind a reverse proxy, enable TLS, and apply rate limiting and authentication.

For more details and advanced configuration options, consult the repository README and configuration files in the project source.