MCP Server Documentation Access for llms.txt Files

Overview

This MCP (Model Context Protocol) server exposes documentation and other text assets to AI agents and tooling so they can retrieve knowledge from local or remote sources. It makes llms.txt files and arbitrary documentation reachable over HTTP, normalizes common discovery patterns, and supports backend storage options such as a local filesystem, Git repositories, or remote URLs. The server is useful when you want agents to access curated context (APIs, docs, design notes) without embedding those assets directly into the model.

By providing a small, predictable HTTP surface and optional manifests, the MCP server reduces friction for retrieval-augmented workflows: agents can discover llms.txt, fetch documents, and stream content for indexing, summarization, or contextual completion. This approach enables incremental updates and consistent access control while keeping integrations simple for developers building AI assistants and tooling.

Features

• Serve llms.txt and other plain-text or markdown files to AI agents
• Support for local filesystem, Git-backed docs, and remote HTTP sources
• Discovery endpoints (manifest / well-known) for automatic agent discovery
• Optional access control and CORS configuration for hosted deployments
• Docker-friendly and minimal dependency install for development
• Simple HTTP API that returns raw content and metadata (size, mime, updated)

Installation / Configuration

Clone the repository and run with Docker or locally. Example commands below assume a Unix-like environment.

Clone:

git clone https://github.com/da1z/docsmcp.git
cd docsmcp

Run with Docker:

# build and run
docker build -t docsmcp .
docker run -it --rm -p 8080:8080 \
  -e MCP_PORT=8080 \
  -e MCP_STORAGE_PATH=/data/docs \
  -v $(pwd)/example-docs:/data/docs \
  docsmcp

Run locally (Python/Node runtime may vary; this demonstrates a generic entry):

# install deps (example for a Python/uvicorn-based server)
pip install -r requirements.txt
export MCP_PORT=8080
export MCP_STORAGE_PATH=./example-docs
uvicorn app:app --host 0.0.0.0 --port $MCP_PORT

Configuration environment variables (common):

API examples (assumes server at http://localhost:8080):

# discover server manifest (well-known entry)
curl http://localhost:8080/.well-known/mcp

# fetch repository llms.txt (served at root)
curl http://localhost:8080/llms.txt

# fetch a specific file (markdown, text)
curl http://localhost:8080/files/path/to/file.md

Available Resources

The MCP server exposes a small set of resources useful to agents and developers:

• /.well-known/mcp — JSON manifest describing available contexts and endpoints (discovery)
• /llms.txt — combined or repository-level llms.txt used to declare agents’ instructions or allowed resources
• /files/{path} — raw file content (supports markdown, text, txt, code)
• Optional storage backends:
  • Local filesystem directory
  • Git repository clone (periodic sync)
  • Remote HTTP/HTTPS URLs (proxied or indexed)
  • Cloud object storage (S3-compatible) via configuration

Sample .well-known/mcp response (illustrative):

{
  "name": "example-docs",
  "version": "1.0",
  "llms_txt": "/llms.txt",
  "files_root": "/files/",
  "sources": [
    "local:/data/docs",
    "git:https://github.com/example/repo"
  ]
}

Sample llms.txt:

# llms.txt - allowed and recommended contexts
Allow: /files/specs/*
Deny: /files/private/*
Priority: /files/overview.md

Use Cases

• Agent context retrieval

  • An assistant can use the /.well-known/mcp manifest to discover where llms.txt lives and then fetch /llms.txt to learn which document paths are safe or high-priority for answering user queries.

• RAG (Retrieval-Augmented Generation) ingestion

  • Periodically pull /files/* content to index into a vector store. The manifest can enumerate remote Git or HTTP sources to include in the ingestion pipeline.

• Automated documentation summarization

  • A job fetches the latest docs via /files and runs summarizers to produce concise changelogs or release notes for models to reference.

• Secure shared knowledge for multi-agent systems

  • Use CORS and access controls to limit which agents or services can fetch sensitive context while still exposing public docs.

Concrete example — fetch and index:

import requests
base = "http://localhost:8080"
manifest = requests.get(f"{base}/.well-known/mcp").json()
llms = requests.get(f"{base}/llms.txt").text
content = requests.get(f"{base}/files/guide/getting-started.md").text
# pass 'content' to vectorizer / tokenizer / indexer

Notes for Developers

• Keep llms.txt authoritative: agents commonly use