Parallel Task MCP Server for Deep Research Tasks

Overview

The Parallel Task MCP (Model Context Protocol) Server is a lightweight orchestration layer designed to run many research-oriented tasks in parallel while exposing a simple HTTP API for submission, monitoring, and retrieval. It targets workflows that benefit from batching, concurrency, and progress tracking — for example, multi-query literature reviews, large-scale prompt evaluations, or distributed scraping and extraction jobs.

By grouping work into tasks and delegating execution to a pool of workers, the MCP server helps teams accelerate experiments and keep stateful visibility into long-running jobs. It provides mechanisms to queue jobs, stream intermediate results, retry failed subtasks, and fetch consolidated outputs, making it suitable as a backbone for research tooling and automation pipelines.

Features

• Parallel task execution with worker pooling and concurrency controls
• Batch submission of multiple subtasks in a single request
• Real-time progress and status endpoints for monitoring
• Result aggregation and streaming for intermediate outputs
• Retry, timeout, and failure-handling policies per task
• Simple REST API suitable for integration with scripts, UIs, or other services
• Optional Docker and environment-based configuration

Installation / Configuration

Clone the repository and start the server with either Docker (recommended for reproducible environments) or a local runtime.

Using Git + Docker:

git clone https://github.com/parallel-web/task-mcp.git
cd task-mcp
docker-compose up --build

If the project provides a Node or Python runtime, a typical local workflow looks like:

# Node (example)
npm install
npm run start

# Python (example)
pip install -r requirements.txt
python server.py

Environment variables commonly used to configure the server:

MCP_PORT=8080              # HTTP port
MCP_CONCURRENCY=8         # Max concurrent workers
MCP_TASK_TIMEOUT=300      # Task timeout in seconds
MCP_PERSISTENCE=memory    # persistence driver: memory, sqlite, redis
MCP_LOG_LEVEL=info

Sample docker-compose excerpt:

version: "3.8"
services:
  mcp-server:
    build: .
    ports:
      - "8080:8080"
    environment:
      - MCP_PORT=8080
      - MCP_CONCURRENCY=8
      - MCP_PERSISTENCE=redis
      - REDIS_URL=redis://redis:6379
  redis:
    image: redis:alpine

Available Resources

Below are common REST endpoints and their purposes. Exact routes may vary; check the server’s API docs or OpenAPI spec if shipped in the repo.

Task payload (typical JSON):

{
  "title": "Cite extraction for AI safety papers",
  "work": [
    {"input": "paper_url_or_text_1", "params": {"model":"gpt-x", "timeout":30}},
    {"input": "paper_url_or_text_2", "params": {"model":"gpt-x"}}
  ],
  "concurrency": 4,
  "retry_policy": {"retries":2, "backoff_seconds":5}
}

Use Cases

1. Literature review across hundreds of papers

   • Submit a batch where each subtask asks an LLM to summarize a paper’s contributions, then stream intermediate summaries back to a dashboard as workers complete them. Use concurrency to limit API costs.

2. Prompt/Model comparison experiments

   • Run the same set of prompts across multiple model adapters in parallel. Aggregate outputs and compute metrics to compare model behavior or latency.

3. Large-scale metadata extraction

   • Crawl a list of URLs, send page text to MCP subtasks for structured extraction (authors, dates, citations). Use retry and timeout settings to handle flaky sources.

4. Data labeling autopilot

   • Use an MCP-backed workflow to generate candidate labels with models, let human reviewers sample streamed results, and restart or refine failed or low-confidence subtasks.

Concrete example: submit tasks with curl

curl -X POST http://localhost:8080/tasks \
  -H "Content-Type: application/json" \
  -d '{
    "title":"Keyword extraction",
    "work":[
      {"input":"Text A"},
      {"input":"Text B"}
    ],
    "concurrency":2
  }'

The server returns a task id. Poll status:

curl http://localhost:8080/tasks/<TASK_ID>

Stream intermediate outputs (if supported):

curl http://localhost:8080/tasks/<TASK_ID>/stream

Tips for Developers

• Start small: use low concurrency locally before scaling to Redis-backed persistence and many workers.
• Instrument tasks with unique ids and metadata to make debugging easier in logs and dashboards.
• Use streaming endpoints for UIs that need progressive updates rather than waiting for a final result.
• Configure sensible retry and timeout policies for external API calls within workers to avoid cascading failures.

This MCP server is intended as a practical orchestration layer for research and batch workflows where parallelism, visibility, and repeatability matter. Integrate it as the control plane for experiments and pipelines to reduce time-to-insight.