Olostep MCP Server: Web Scraping and Content Extraction

Overview

Olostep MCP Server implements the Model Context Protocol (MCP) focused on web scraping and content extraction. It provides reliable, geo-aware scraping with JavaScript rendering and structured outputs designed to be consumed by language models or downstream tools. The server extracts clean Markdown from web pages, returns structured search results, and can produce sitemaps for crawled sites — all as MCP-compatible context blocks.

This server is useful when you need repeatable, low-cost extraction of high-quality text for indexing, summarization, or LLM prompting. Key capabilities include rendering client-side JavaScript, routing requests through region-specific endpoints for localized results, and returning sanitized Markdown to reduce noise and token waste in model inputs.

Features

• Extract clean, readable Markdown from arbitrary web pages
• Produce structured search results (rank, snippet, URL, metadata)
• Generate sitemaps and link graphs for a given domain
• JavaScript rendering for SPA and dynamic sites (headless browser)
• Geo-routing / localization parameters for region-specific scraping
• Caching, rate-limiting, and configurable concurrency for cost control
• MCP-compatible JSON outputs for easy LLM integration
• Optional Docker deployment and environment-configurable renderer

Installation / Configuration

Clone and run the server locally or via Docker. Below are example workflows.

Clone repository

git clone https://github.com/olostep/olostep-mcp-server.git
cd olostep-mcp-server

Run with Docker

# Run a production-ready container (image name is illustrative)
docker pull olostep/olostep-mcp-server:latest
docker run -p 8080:8080 \
  -e PORT=8080 \
  -e RENDERER_URL=http://renderer:3000 \
  -e GEO_DEFAULT=us \
  olostep/olostep-mcp-server:latest

Docker Compose example

version: "3.8"
services:
  mcp:
    image: olostep/olostep-mcp-server:latest
    ports: ["8080:8080"]
    environment:
      - PORT=8080
      - RENDERER_URL=http://renderer:3000
      - CACHE_TTL=3600
  renderer:
    image: olostep/headless-renderer:latest
    ports: ["3000:3000"]

Environment variables (common)

PORT=8080                 # HTTP server port
RENDERER_URL=http://...   # Headless renderer service (Playwright/Puppeteer)
CACHE_TTL=3600            # Seconds to cache responses
GEO_DEFAULT=us            # Default geo/location tag for requests
RATE_LIMIT=10             # Requests per second

Available Resources

The server exposes MCP-style HTTP endpoints that return structured JSON. Typical endpoints:

Example extract request

curl -X POST "http://localhost:8080/mcp/extract" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com/article",
    "render": true,
    "geo": "gb"
  }'

Example extract response (abbreviated)

{
  "title": "Example Article",
  "url": "https://example.com/article",
  "markdown": "## Heading\n\nCleaned content...",
  "meta": {"author":"Jane Doe","published":"2024-01-15"},
  "links": ["https://example.com/about", "..."]
}

Geo-routing and rendering options

• render: true/false — enable JS rendering
• geo: ISO country code or region string — ask the server to perform region-aware requests
• depth, maxPages — sitemap crawl limits

Use Cases

1. Feeding a language model with clean page context

• Problem: LLMs receive noisy HTML and get charged for unnecessary tokens.
• Approach: Call /mcp/extract with render=true; supply the resulting markdown as the model context.
• Example:

curl -X POST http://localhost:8080/mcp/extract -H "Content-Type:application/json" \
  -d '{"url":"https://news.example/article","render":true}'

2. Localized SERP or storefront scraping

• Problem: Need search results as seen from a specific country.
• Approach: Use /mcp/search with geo=“fr” and query parameters to obtain region-specific results.
• Example:

curl -X POST http://localhost:8080/mcp/search -H "Content-Type:application/json" \
  -d '{"query":"best coffee shops","site":"example.com","geo":"fr"}'

3. Site migration / audit with sitemaps

• Problem: Identify pages, detect missing metadata, and plan redirects.
• Approach: Run /mcp/sitemap with domain and depth limits to get a structured list of pages and last modification times.
• Example:

curl -X POST http://localhost:8080/mcp/sitemap -H "Content-Type:application/json" \
  -d '{"domain":"example.com","depth":3,"maxPages":1000}'

4. Building a knowledge base or search index

• Use extract to get normalized Markdown and metadata, then store into your vector DB or index for retrieval-augmented generation.

Links and Further Reading

• Source code and issues: https://github.com/olostep/olostep-mcp-server
• Consider pairing with a headless-renderer (Playwright/Puppeteer) for best JS-rendered results.

If you plan to deploy at scale, configure caching, rate limits, and a dedicated headless-renderer cluster to balance cost and performance.