Web Search MCP Server: Summaries & Page Extraction

Overview

The Web Search MCP Server enables Local LLMs to perform live web search, extract page content, and produce concise, citation-aware summaries via a Model Context Protocol (MCP)–compatible toolset. It sits between a local or private LLM and the open web: when the LLM needs up-to-date or source-specific information, it calls the MCP server to retrieve search results, fetch and parse pages, and return focused content suitable for model context windows.

This approach helps local deployments avoid embedding a full web crawler into the model while still benefiting from current web data. By returning cleaned text, chunked context, and short summaries with source links, the server simplifies retrieval-augmented generation workflows and improves factual grounding for downstream LLM responses.

Features

• Expose MCP-compatible tools for:
  • Web search (query -> ranked results)
  • Page extraction (HTML -> cleaned text and metadata)
  • Summarization (long text -> concise digest with sources)
  • Chunking (split large pages for context windows)
• Configurable search backend and scraping behavior
• Rate limiting, caching, and user-agent customization
• Returns structured JSON including source URLs and timestamps
• Lightweight: runs as a single process or in Docker for easy local deployment

Installation / Configuration

Basic steps to get the server running locally:

1. Clone the repo and install dependencies:

git clone https://github.com/mrkrsl/web-search-mcp.git
cd web-search-mcp
# Example for a Node-based project
npm install

2. Create a .env or set environment variables (example variables shown):

PORT=8080
SEARCH_API_KEY=your-search-api-key   # optional (for paid search APIs)
SEARCH_PROVIDER=serpapi|bing|google  # provider selection if supported
USER_AGENT="web-search-mcp/1.0"
CACHE_TTL=3600
MAX_RESULTS=5
CHUNK_SIZE=1000

3. Run the server:

# Run locally
npm start

# Or with Docker
docker build -t web-search-mcp .
docker run -p 8080:8080 \
  -e SEARCH_API_KEY="$SEARCH_API_KEY" \
  -e PORT=8080 \
  web-search-mcp

Adjust configuration to point at a preferred search API or to enable direct scraping. When using third-party search services, supply the provider key as shown above.

Available Tools / Resources

The server exposes a small set of MCP-compatible tools (logical names below). Each tool returns JSON designed to be easy to consume by LLM orchestration code.

Example endpoint mapping (may vary by implementation):

• POST /tools/web.search
• POST /tools/page.extract
• POST /tools/page.summarize

Example request (search):

curl -X POST "http://localhost:8080/tools/web.search" \
  -H "Content-Type: application/json" \
  -d '{"query":"latest AI model evaluation benchmarks", "max_results":5}'

Example response (search):

{
  "results": [
    {"title":"Model Evaluation 2026", "url":"https://example.com/eval", "snippet":"Overview of benchmarks..."},
    ...
  ],
  "query":"latest AI model evaluation benchmarks",
  "timestamp":"2026-04-10T12:34:56Z"
}

Use Cases

• Retrieval-Augmented Generation (RAG)
  • A local LLM calls web.search to find relevant pages, page.extract to retrieve content, and page.summarize to produce a 3–5 sentence answer with citations that the model can include verbatim in its response.
• Fact-checking and citation
  • Use page.extract to pull authoritative text, then feed the extracted passages to a fact-checking pipeline or comparator model.
• Context window management
  • For long pages, page.chunks produces appropriately sized chunks with offsets, enabling the LLM to include only the most relevant chunks in prompt context.
• Multi-step tool chains
  • Compose search -> extract -> summarize as discrete MCP tool calls from the model, keeping each step auditable and reducing hallucination risk.

Quick Integration Example (Python)

import requests

base = "http://localhost:8080"
resp = requests.post(f"{base}/tools/web.search", json={"query":"climate policy 2026", "max_results":3})
results = resp.json()["results"]

# Extract and summarize first result
url = results[0]["url"]
extract = requests.post(f"{base}/tools/page.extract", json={"url": url}).json()
summary = requests.post(f"{base}/tools/page.summarize", json={"text": extract["text"], "max_length": 120}).json()
print(summary["summary"])

Notes and Best Practices

• Respect robots.txt and copyright: configure scraping behavior and caching duration to comply with site policies.
• Rate limit search and scraping to avoid API bans; prefer an API-backed search provider for production workloads.
• Cache frequently-requested pages to reduce load and accelerate LLM workflows.
• Tune chunk size based on your LLM’s context window to maximize useful context while minimizing token cost.

For the source and latest updates, see the project repository: https://github.com/mrkrsl/web-search-mcp.