RAG Web Browser MCP Server for Markdown Scraping

Overview

This MCP (Model Context Protocol) server provides a lightweight HTTP tool layer that the Apify RAG Web Browser (retrieval-augmented generation) can call to perform web searches, fetch pages, and convert web content to cleaned Markdown. Instead of returning raw HTML or full page dumps, the server extracts readable content, removes boilerplate, and produces Markdown suitable for downstream indexing, prompt context, or LLM tool usage.

The server implements a set of “tools” accessible over simple HTTP endpoints (compatible with MCP tool invocation patterns). This makes it easy to integrate web retrieval and scraping into RAG workflows: the model can call a search tool, pick candidate URLs, and call a scrape tool that returns cleaned Markdown and metadata ready for embedding or summarization.

Features

• Convert web pages and URLs to cleaned, readable Markdown
• Perform web searches and return structured search results (title, url, snippet)
• Optionally use a headless browser (for JavaScript-heavy pages) or a lightweight fetch path
• Return metadata alongside content: title, canonical URL, retrieval timestamp, word/char counts
• Simple HTTP API designed for MCP-style tool invocation by agent frameworks
• Docker-friendly and configurable via environment variables

Installation / Configuration

Clone and run locally or in Docker. Typical options:

Clone and install

git clone https://github.com/apify/mcp-server-rag-web-browser.git
cd mcp-server-rag-web-browser
npm install
npm run start

Run with Node (example with environment variables)

PORT=3000 USE_BROWSER=false LOG_LEVEL=info npm run start

Run with Docker

docker build -t mcp-rag-web-browser .
docker run -p 3000:3000 \
  -e PORT=3000 \
  -e USE_BROWSER=true \
  -e MAX_CONTENT_BYTES=500000 \
  mcp-rag-web-browser

Common configuration environment variables

• PORT: HTTP port (default: 3000)
• USE_BROWSER: “true” to use a headless browser (Puppeteer) for fetching JS-driven pages; “false” to use simple fetch
• MAX_CONTENT_BYTES: max number of bytes to fetch/scrape
• ALLOWED_ORIGINS / CORS configuration
• LOG_LEVEL: debug/info/warn/error

Available Tools / Resources

The server exposes a concise set of endpoints that map to MCP-style tools. Example endpoints:

Typical scrape response

{
  "markdown": "# Page title\n\nContent in markdown...",
  "text": "Plain text fallback",
  "metadata": {
    "url": "https://example.com/article",
    "title": "Article title",
    "domain": "example.com",
    "retrievedAt": "2026-04-09T12:00:00Z",
    "contentLength": 34567
  }
}

Implementation notes

• The server strips navigation, ads, and common boilerplate using a readability/content extraction step before converting to Markdown.
• If USE_BROWSER is true, it runs a headless Chromium instance to render dynamic pages and capture the final DOM prior to extraction.
• Search results are provided via a configurable search backend (could be a public search API or custom index).

Use Cases

• RAG context augmentation: Let a model call the search tool to find candidate pages and call the scrape tool to embed cleaned Markdown into prompt context or vector stores.
• Building a web content indexer: Batch URLs, scrape them to Markdown, and send the resulting content to your embedding/indexing pipeline.
• Automated QA and summarization: Fetch articles in real time, convert to Markdown, and feed into an LLM for summarization or question answering.
• Content auditing and compliance: Convert pages to a consistent Markdown format for easier diffing, metadata checks, or policy scanning.

Concrete example (RAG flow)

1. Agent asks the MCP server: POST /tools/search { “q”: “latest climate policy summary”, “limit”: 3 }
2. Agent selects a URL and calls POST /tools/scrape { “url”: “https://example.gov/policy” }
3. The server returns cleaned Markdown and metadata; the agent adds that Markdown to the retrieval context for the LLM.

Troubleshooting & Tips

• If pages return incomplete content, enable USE_BROWSER to capture client-rendered DOM.
• Watch MAX_CONTENT_BYTES and MAX_HTML_DEPTH settings to avoid very large downloads.
• For high throughput, run behind a process manager or container orchestration and scale browser instances carefully (headless browsers are resource-heavy).
• Enable request logging (LOG_LEVEL=debug) to inspect pipeline steps: fetch -> extract -> markdown conversion.

Repository and source code

• Source and examples: https://github.com/apify/mcp-server-rag-web-browser

This MCP server is intended to be a practical tool bridge between web retrieval and RAG agents — providing predictable, scrubbed Markdown that plays well with LLM workflows and indexing systems.