Overview

This MCP (Model Context Protocol) server provides a practical way to make very large files available to LLMs and other downstream systems without loading the entire file into memory. It splits files into indexed, navigable chunks, supports streamed delivery of chunked content, and supplies a small, in-memory LRU cache to keep recently used chunks hot. The result is reduced latency for repeated reads, predictable memory usage, and improved throughput for batching or streaming workloads.

Beyond chunking and streaming, the server exposes lightweight regex-powered search across file chunks and metadata, enabling quick navigation to relevant sections before feeding content to a model. This is especially useful when you only need specific passages (e.g., code snippets, paragraphs, timestamps) rather than the whole file.

Features

Smart chunking: configurable chunk size with overlap to preserve context boundaries
Streamed delivery: chunk-by-chunk HTTP streaming for progressive consumption
Navigation APIs: indexed access to chunks and basic metadata for efficient context assembly
Regex-powered search: fast search over chunked content using regular expressions
LRU caching: in-memory Least Recently Used cache for hot chunks to reduce disk I/O
Simple configuration: JSON/YAML-style options for chunk size, cache limits, and search behavior
Suitable for large files (multi-GB): avoids full-file reads and keeps memory bounded

Installation / Configuration

Clone the repository and run via Docker, or run locally if you prefer.

Clone repository:

git clone https://github.com/willianpinho/large-file-mcp.git
cd large-file-mcp

Run with Docker (recommended for production-like isolation):

# build image
docker build -t large-file-mcp .

# run, map storage and expose port 8080
docker run -d \
  -p 8080:8080 \
  -v /path/to/your/files:/data/files \
  -e CONFIG_PATH=/data/files/mcp-config.json \
  --name large-file-mcp \
  large-file-mcp

Run locally (node example):

# install dependencies and start
npm install
npm start
# or, for dev:
npm run dev

Example configuration (mcp-config.json):

{
  "storage_path": "/data/files",
  "chunk_size": 64,
  "chunk_overlap": 8,
  "cache_max_entries": 256,
  "enable_search": true,
  "max_stream_rate_kb_s": 0
}

chunk_size: size in KB for each chunk
chunk_overlap: overlap in KB between consecutive chunks to preserve context
cache_max_entries: number of chunk entries kept in LRU cache
max_stream_rate_kb_s: 0 for unlimited, otherwise throttle streaming

Available Resources

Typical HTTP resources and how to use them (names are representative — check your deployed server’s OpenAPI for exact routes):

GET /files
- Returns list of ingested files with IDs and metadata.
- Example: curl http://localhost:8080/files
GET /files/:fileId/meta
- Returns chunk count, sizes, and index information for a file.
GET /files/:fileId/chunks/:index
- Returns a single chunk by index. Useful for random access and deterministic context assembly.
- Example:
```
curl http://localhost:8080/files/abcd1234/chunks/10
```
GET /files/:fileId/stream
- Streams file chunks sequentially (chunked transfer); clients can consume progressively.
- Example:
```
curl -N http://localhost:8080/files/abcd1234/stream
```
POST /search
- Body: { “fileId”: “…”, “pattern”: “TODO|fixme”, “maxResults”: 10 }
- Returns matching chunk indices and excerpts. Supports regex for fine-grained navigation.
POST /ingest
- Upload a new file for server-side chunking/indexing.

These resources pair well with an MCP client that requests a subset of chunks to build context windows for a model call.

Use Cases

Feeding LLMs with focused context: Search for the paragraph that answers a question, fetch only the chunk(s) needed, and send them as prompt context instead of the full file.
Streaming transcripts: Deliver long audio transcripts progressively to a transcription or summarization model while keeping memory low.
Large-code search and retrieval: Index huge codebases and use regex search to find relevant functions or classes by pattern, then stream matching chunks to an assistant.
Video subtitle and metadata delivery: Chunk subtitle files with overlap and stream time-aligned text for downstream inference or real-time inspection.
Embeddings pipeline: Materialize only the chunks you need to embed (or re-embed) and use LRU caching to avoid repeated disk reads during batch embedding jobs.

How LRU caching helps

The server uses an LRU cache tuned by cache_max_entries. When a client requests a chunk, the server:

Checks the cache for the chunk key (fileId + chunkIndex).
Returns from cache if present (fast, RAM-only).
If absent, reads the chunk from disk, inserts into the cache, and returns it.

This keeps memory bounded while improving latency for hot reads and typical access patterns (e.g., sequential streaming, repeated searches).

Tips and Best Practices

Choose a chunk size that balances context preservation and efficiency — 32–128 KB is a common starting point for text.
Use chunk_overlap to avoid cutting sentences or code examples in half (8–16 KB typical).
Tune cache_max_entries according to available RAM and concurrent workload.
Enable regex search only when needed; complex expressions can be CPU intensive on large corpora.
For production, run behind a reverse proxy and expose only necessary endpoints; add authentication and rate limiting as appropriate.

For more details, consult the repository’s README and OpenAPI spec in the project for exact routes and schema definitions.

Large File MCP Server: Chunking, Streaming, LRU Caching