WebScraping.AI MCP Server for Web Data Scraping

Overview

The WebScraping.AI MCP Server provides a standardized, programmatic way to extract page content, run JavaScript, and capture rendered artifacts (HTML, screenshots, HARs) from websites for use by downstream systems or model-driven agents. It implements the Model Context Protocol (MCP) pattern so LLMs and automation agents can request web interactions as discrete “tools” and receive structured results without embedding scraping logic inside the model.

This server is useful when you need reliable, repeatable web data extraction at scale — for example, enriching an LLM prompt with live page content, building extraction pipelines that handle client-rendered sites, or running browser-driven audits. It decouples scraping infrastructure from application code, adds concurrency and resource control, and exposes a simple HTTP API that agents or services can call.

Features

• MCP-compatible interface for model/tool integrations
• Headless browser rendering (JavaScript execution) for SPA and dynamic pages
• HTML extraction, DOM querying, and text/attribute extraction
• Screenshots and HAR capture for debugging and full-request traces
• Concurrent job processing with queueing and resource limits
• Proxy and session management for rotation and isolation
• Retry, timeout, and error handling policies for robust scraping
• Docker-ready for containerized deployment and Kubernetes-friendly operation
• Configurable logging and metrics for observability

Installation / Configuration

Quickstart (clone and run locally)

# Clone the repository
git clone https://github.com/webscraping-ai/webscraping-ai-mcp-server.git
cd webscraping-ai-mcp-server

# Install dependencies and start (Node.js project)
npm ci
npm run build
npm start

Run with Docker

# Build image locally
docker build -t webscraping-ai-mcp-server .

# Run container (example)
docker run -p 3000:3000 \
  -e PORT=3000 \
  -e HEADLESS=true \
  -e CONCURRENCY=4 \
  webscraping-ai-mcp-server

Docker Compose example

version: "3.8"
services:
  mcp-server:
    image: webscraping-ai/mcp-server:latest
    ports:
      - "3000:3000"
    environment:
      PORT: 3000
      CONCURRENCY: 6
      HEADLESS: "true"
      REDIS_URL: "redis://redis:6379"
  redis:
    image: redis:7

Common environment variables (example — check repo for exact names)

Tip: Always consult the repository README for exact environment names and advanced configuration options (provisioning browser binaries, auth secrets, or custom tool plugins).

Available Resources

The server exposes a small set of MCP-style tools (HTTP endpoints) that map to common web interactions. Typical resources you can expect:

• /mcp or /tools (MCP request endpoint) — accept MCP-formatted tool calls
• fetch/render — navigate to a URL and return rendered HTML
• evaluate — run arbitrary JS in the page context and return result
• screenshot — capture PNG or JPEG of the rendered page
• har — capture network HAR for the navigation
• query — run CSS/XPath queries and return matches or structured extracts
• cookies/headers — read or set session cookies and headers

Note: Exact endpoint names and request schemas may differ by version; use the repo docs or OpenAPI spec (if included) for precise request/response shapes.

Use Cases

1. LLM-assisted browsing and summarization

   • An agent calls the MCP server to fetch a product page (rendered HTML) and returns cleaned text for the model to summarize or answer questions about live content.

2. Structured data extraction pipeline

   • Use query tools to extract product titles, prices, and availability from a set of URLs. Send results