CO
Developer Tools

Context Crystallizer MCP Server for AI Context Engineering

Transform large repositories into crystallized AI-consumable knowledge with the Context Crystallizer MCP server via systematic analysis and optimization.

View on GitHub

Overview

Context Crystallizer is an MCP (Model Context Protocol) server designed to convert large codebases and document repositories into tightly scoped, AI-consumable context windows. It performs systematic analysis — file discovery, token-aware chunking, embedding generation, and scoring — producing “crystallized” context fragments that maximize relevance for retrieval-augmented generation (RAG), in-context learning, and prompt engineering.

The server exposes a standard MCP-compatible REST API and CLI to ingest repositories, run optimization pipelines, and serve ranked context to models. It integrates with embedding providers and vector databases so you can plug it into your existing LLM workflows and use cases such as code summarization, automated QA, or building compact knowledge bases for multi-file prompts.

Features

  • Repository and filesystem ingestion (local or remote)
  • Token-aware chunking and overlap control to respect model limits
  • Embedding generation via pluggable providers (OpenAI, Hugging Face, etc.)
  • Vector store integration (Milvus, Weaviate, Pinecone, local FAISS)
  • Context scoring and optimization to select high-utility fragments
  • MCP-compatible REST API and simple CLI tooling
  • Configurable pipelines for custom heuristics and filtering
  • Output formats: JSON, MCP fragments, or ready-to-insert prompt snippets
  • Batch processing and incremental updates for large repos

Installation / Configuration

Requirements:

  • Node.js 18+ (or use Docker)
  • Access to an embedding provider and optional vector DB
  • Git (for cloning the repo)

Clone and install:

git clone https://github.com/hubertciebiada/context-crystallizer.git
cd context-crystallizer
npm install

Run locally (development):

# Start server with .env file present
npm run dev

Docker:

# Build image
docker build -t context-crystallizer .

# Run container with environment variables
docker run -p 8080:8080 \
  -e EMBEDDING_PROVIDER=openai \
  -e OPENAI_API_KEY=sk-... \
  -e VECTOR_DB=local \
  context-crystallizer

Example .env configuration:

PORT=8080
EMBERDING_PROVIDER=openai
OPENAI_API_KEY=sk-...
VECTOR_DB=faiss
CHUNK_TOKENS=800
CHUNK_STRIDE=128
MAX_CONTEXT_TOKENS=3000

Configuration options (common):

  • CHUNK_TOKENS: target tokens per fragment
  • CHUNK_STRIDE: overlap tokens
  • EMBEDDING_PROVIDER: provider key
  • VECTOR_DB: local | faiss | pinecone | milvus
  • MAX_CONTEXT_TOKENS: target total tokens to return for a query

Available Tools

  • CLI
    • ingest: scan a repo and produce embeddings
    • optimize: run heuristics to condense context
    • serve: start MCP REST API
  • REST API (MCP endpoints)
    • /ingest — POST: trigger repository ingestion
    • /fragments — GET: list stored fragments
    • /search — POST: retrieve ranked fragments for a query
    • /optimize — POST: run context optimization for a query
  • Connectors
    • Embeddings: OpenAI, Hugging Face
    • Vector DBs: FAISS (local), Pinecone, Weaviate, Milvus
  • Optional Web UI for browsing fragments and manual curation

API endpoints summary:

EndpointMethodPurpose
/ingestPOSTIngest files or repo URL
/searchPOSTRetrieve ranked context fragments
/optimizePOSTRun optimization and return a crystallized context
/fragmentsGETList stored fragments and metadata

Example ingest (curl):

curl -X POST http://localhost:8080/ingest \
  -H "Content-Type: application/json" \
  -d '{"repo_url":"https://github.com/your-org/your-repo","branch":"main"}'

Use Cases

  1. Generate compact context for an LLM (RAG / prompt composition)
    • Ingest a repository, then request /search with a user question. The MCP server returns a ranked sequence of fragments sized to fit your target model window. Use the returned fragments as the prompt context for your LLM.
    • Example query:
curl -X POST http://localhost:8080/search \
  -H "Content-Type: application/json" \
  -d '{"query":"How does authentication work in the project?","max_context_tokens":2000}'

  1. Codebase summarization and onboarding

    • Run ingest + optimize, then call /optimize to produce a condensed narrative describing key modules, architecture, and entry points. Useful for generating README updates or onboarding docs.

  2. QA over documentation

    • Index large docs and tutorials; answer user questions by retrieving the most relevant crystallized fragments rather than full documents, reducing noise and improving answer relevance.

  3. Prompt engineering and test-case generation

    • Use the server’s fragment scoring to surface the most test-relevant code blocks or API snippets for automated test generation or few-shot prompt construction.

  4. Incremental updates for CI workflows

    • Integrate ingest into a CI job to re-index changed files and keep the crystallized context up-to-date as the repository evolves.

Tips and Best Practices

  • Tune CHUNK_TOKENS and CHUNK_STRIDE to balance fragment granularity and retrieval precision.
  • Use a persistent vector DB for production workloads; local FAISS is good for experimentation.
  • Combine keyword filtering and semantic search to reduce false positives for domain-specific codebases.

For full implementation details, examples, and extension points, see the project repository: https://github.com/hubertciebiada/context-crystallizer

Tags:developer-tools