Defuddle Fetch MCP Server: Clean Markdown Extraction

Overview

Defuddle Fetch MCP Server is a lightweight HTTP server that fetches arbitrary web pages and converts them into clean, structured Markdown suitable for documentation, content migration, and static-site generation. It combines conservative HTML parsing with content-aware normalization to produce Markdown that preserves semantic structure (headings, lists, code blocks, links, images) while stripping noisy site chrome, inline styles, and trackers that typical HTML-to-Markdown converters often retain.

The server is intended for developers and content engineers who need reproducible, automatable content extraction workflows. It exposes a simple HTTP API so it can be integrated into pipelines, build systems, or run as a single-service content transformation step before indexing, archiving, or importing into CMS/static-site sources.

Features

• Fetch remote web pages and convert to Markdown in a single request
• Heuristics to remove navigation, ads, and peripheral site chrome
• Preserve semantic elements: headings, lists, tables, code blocks, blockquotes, inline formatting
• Optional frontmatter generation with source metadata (URL, title, fetch timestamp)
• Asset handling: rewrite or inline image URLs, optionally download assets for local hosting
• Configurable sanitization and allowed HTML elements
• Simple HTTP API suitable for integration into CI/CD, static site migrations, or archiving tools
• Rate limiting, caching, and domain allowlists (configurable)
• Docker-friendly and scriptable via curl or HTTP clients

Installation / Configuration

Clone the repository and run the server locally or inside a container. The exact commands depend on the language/runtime in the repository; the examples below show common patterns.

Clone repository:

git clone https://github.com/domdomegg/defuddle-fetch-mcp-server.git
cd defuddle-fetch-mcp-server

Run locally (Node.js-style example):

# install deps
npm install

# run (default port 3000)
npm start

Run with Docker:

# build
docker build -t defuddle-fetch-mcp-server .

# run
docker run -p 3000:3000 \
  -e PORT=3000 \
  -e CACHE_TTL=3600 \
  -e ALLOWLIST_DOMAINS=example.com,another.com \
  defuddle-fetch-mcp-server

Configuration via environment variables (typical options):

PORT=3000              # HTTP port
CACHE_TTL=3600         # seconds to cache converted pages
ALLOWLIST_DOMAINS=*    # comma-separated list; use * to allow all (use with caution)
FRONTMATTER=true       # include YAML frontmatter with metadata
ASSET_MODE=copy        # copy | rewrite | inline
MAX_CONCURRENCY=5      # concurrent fetch/conversion jobs

If the project provides a configuration file, you can set the same options in JSON/YAML and mount it into the container or pass a path at service start-up.

API Reference

Common endpoints (example names; confirm actual routes in the server source):

Example GET request:

curl "http://localhost:3000/convert?url=https://example.com/article/123"

Example POST request with options:

curl -X POST http://localhost:3000/convert \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com/article/123",
    "options": {
      "frontmatter": true,
      "assetMode": "rewrite"
    }
  }'

Response: plain Markdown text (Content-Type: text/markdown) or JSON wrapper containing metadata and Markdown payload, depending on server configuration.

Available Resources

• GitHub repository: https://github.com/domdomegg/defuddle-fetch-mcp-server — source, issues, and contribution notes
• Example scripts: look for example curl invocations and sample configs in the repo’s examples/ or scripts/ directory
• Integration notes: use the API in build hooks, CI jobs, or as a microservice in front of your static site pipeline

Use Cases

• Static site migration: batch-convert legacy HTML pages into Markdown files with frontmatter for Jekyll/Hugo migration.
• Content archiving: normalize pages to readable Markdown for long-term storage without site-specific CSS/JS noise.
• Newsletter or digest generation: extract article body and images for reuse in email templates.
• Search indexing and NLP preprocessing: convert pages to cleaned Markdown/plaintext for tokenization and semantic processing.
• Automated documentation capture: extract README-like content from web pages and convert to Markdown to include in developer docs or internal knowledge bases.

Tips for Developers

• Use ALLOWLIST_DOMAINS in production to avoid unintended scraping.
• Enable caching (CACHE_TTL) when converting frequently-requested pages to reduce load and improve throughput.
• Test different ASSET_MODE settings to choose between localizing images (copy), rewriting URLs for CDN hosting (rewrite), or embedding small images as base64 (inline).
• Pair this service with an orchestrator or queue (e.g., RabbitMQ, Redis Queue) when converting large batches to control concurrency.

If you need help integrating or want to extend conversion heuristics, the repository contains the conversion logic which you can adapt or override to fit domain-specific HTML patterns.