Scrapling Fetch MCP Server for Bot-Protected Websites

Overview

Scrapling Fetch MCP Server is a small HTTP service that retrieves web pages protected by anti-automation defenses (for example Cloudflare, Akamai, and similar bot mitigations) and returns the page content in HTML or Markdown. It is designed to sit between an LLM-based agent and the web, exposing a simple Model Context Protocol (MCP)-compatible interface so language models can request clean, readable page text without having to handle browser automation or bypass logic themselves.

This server is useful when you need reliable text content from pages that refuse basic HTTP scraping. It encapsulates the complexity of headless-browser rendering, anti-bot handling, and HTML→Markdown conversion and exposes that functionality as an API or as an MCP tool that model-based applications can call directly.

Features

• Fetch fully rendered HTML from bot-protected pages using a headless browser layer
• Convert fetched HTML into Markdown automatically (optional)
• Simple JSON/HTTP API suitable for LLM tool invocation (MCP-compatible)
• Configurable timeouts, concurrency limits, and user-agent options
• Docker image for quick deployment and environment-variable-driven configuration
• Health and metrics endpoints to integrate with orchestration and monitoring

Installation / Configuration

Prerequisites:

• Docker (recommended) or Node/Python runtime (depending on distribution)
• Scrapling account / service credentials if using a hosted anti-bot bypass provider (optional)

Run with Docker (recommended):

# Pull the image and run on port 8080
docker run -d \
  -p 8080:8080 \
  -e SCRAPLING_API_KEY="your_scrapling_api_key" \
  -e FETCH_TIMEOUT=30 \
  --name scrapling-fetch-mcp \
  cyberchitta/scrapling-fetch-mcp:latest

Run from source (example pattern - adapt to language/runtime in repo):

# clone repo
git clone https://github.com/cyberchitta/scrapling-fetch-mcp.git
cd scrapling-fetch-mcp

# install dependencies (example for Node)
npm install

# run with environment variables for configuration
SCRAPLING_API_KEY=your_scrapling_api_key PORT=8080 node server.js

Common environment variables:

Adjust timeouts and concurrency according to your hosting environment and the expected volume of requests.

Available Tools / Resources

The server exposes a small set of HTTP endpoints that are ideal to wrap as MCP tools for LLMs.

• GET /health — returns 200 if server is ready
• GET /metrics — Prometheus-style metrics (if enabled)
• POST /fetch — primary endpoint to fetch a URL and return HTML or Markdown

Example POST /fetch request (JSON):

{
  "url": "https://example.com/article",
  "format": "markdown",
  "render_js": true,
  "timeout": 20
}

Response (success):

{
  "url": "https://example.com/article",
  "status": 200,
  "content_type": "text/markdown",
  "content": "# Article title\n\nArticle text..."
}

Supported query parameters/JSON fields:

• url (string) — required, target URL to fetch
• format (string) — “html” or “markdown” (default: “html”)
• render_js (bool) — whether to execute JS / render client-side content
• timeout (number) — override default per-request timeout (seconds)

If you want to expose the server as an MCP tool for an LLM, provide a tool descriptor that maps to POST /fetch with the required fields (url, format). The server is intentionally minimal so tools remain straightforward.

Use Cases

1. LLM-assisted browsing for research

   • An assistant can call the fetch tool to obtain a clean Markdown version of paywalled or bot-protected article text, then summarize or extract key points without dealing with the page’s obfuscation or JS rendering.

2. Single-page app content capture

   • Fetch dynamic content that requires JS rendering (render_js=true) and convert it to Markdown for downstream parsing and indexing.

3. Content ingestion for indexing or search

   • Use the server to retrieve canonical article text from sites guarded by anti-bot layers and feed the Markdown into a vector store or search index.

4. Compliance and monitoring

   • Periodically fetch protected policy pages or terms-of-service documents using the server, store snapshots (HTML or Markdown), and diff them for changes.

5. Tool integration in LLM workflows

   • Expose the fetch endpoint as an MCP tool so model-based agents can retrieve context-aware page text on demand and combine it with reasoning or data extraction tools.

Examples

Fetch page as Markdown with curl:

curl -X POST http://localhost:8080/fetch \
  -H "Content-Type: application/json" \
  -d '{"url":"https://example.com","format":"markdown","render_js":true}'

Simple health check:

curl http://localhost:8080/health

MCP tool descriptor (example snippet):

{
  "name": "scrapling_fetch",
  "description": "Fetch HTML or Markdown from a URL, including bot-protected pages",
  "endpoint": "http://localhost:8080/fetch",
  "method": "POST",
  "schema": {
    "type": "object",
    "properties": {
      "url": {"type": "string"},
      "format": {"type": "string", "enum": ["html","markdown"]}
    },
    "required": ["url"]
  }
}

Notes and Best Practices

• Respect robots.txt and site terms — using browser-level fetch to bypass bot protections can have legal and ethical implications.
• Cache results where appropriate to reduce repeat headless browser usage and costs.
• Tune concurrency and timeouts to avoid overloading the host or triggering defensive measures on target sites.
• Use API keys and network-level controls to prevent abuse of your deployed instance.