Flyworks Avatar MCP Server Fast Free Zeroshot Lipsync

Overview

The Flyworks Avatar MCP Server implements a simple Model Context Protocol (MCP) endpoint to produce fast, zeroshot lipsync for avatars. It is an open-source, self-hosted service that accepts text or audio and returns time-aligned speech artifacts (audio, viseme timings, and optionally rendered frames or packaged video). The server is designed to minimize setup and dependencies so developers can integrate lipsync into games, virtual production, chatbots, or video pipelines without training per-avatar models.

Zeroshot lipsync here means the server generates viseme or phoneme timing directly from input audio (or synthesized speech) and maps those timings to generic avatar blendshapes or viseme IDs. The MCP server exposes simple REST and WebSocket endpoints so you can request synchronous or streaming outputs, retrieve status, and integrate the timing data into real-time animation systems.

Features

• Fast, zeroshot lipsync: audio-to-viseme timing without per-avatar training
• Supports text-to-speech pipelines and raw audio input
• Returns viseme timings, audio files, and optionally packaged video
• REST API for single-shot requests and WebSocket for streaming/low-latency workflows
• Self-hosted, configurable, with Docker support
• Simple JSON response format compatible with animation systems and game engines
• Minimal server API surface to integrate into existing pipelines

Installation / Configuration

Prerequisites: Node.js (>=16), Docker (optional), and appropriate model files if using local TTS or ASR backends.

Clone and run locally:

git clone https://github.com/Flyworks-AI/flyworks-mcp.git
cd flyworks-mcp
npm install
# Start in development
npm start

Start with Docker:

# Build image
docker build -t flyworks-mcp .
# Run with environment variables (see table below)
docker run -p 8080:8080 \
  -e PORT=8080 \
  -e USE_GPU=false \
  -e MODEL_DIR=/models \
  -v /local/models:/models \
  flyworks-mcp

Environment variables (examples):

Configuration may also be provided by a config.json in the repository root. See the repo for exact keys accepted.

Available Resources

• GitHub repository: https://github.com/Flyworks-AI/flyworks-mcp — main source, issues, and examples
• Example clients: /examples directory (HTTP and Node.js clients)
• Demo HTML: simple browser demo that shows timing data applied to a WebGL avatar (if included in the repo)
• Logs and metrics: /logs output and a /metrics endpoint for basic health and usage stats

API Overview

Common endpoints (example surface; check repo for current HTTP paths):

• GET /health — basic liveness check
• POST /v1/lipsync — generate visemes from audio or text
• POST /v1/say — generate speech audio (TTS) and viseme timings (text input)
• POST /v1/render — optional: return packaged video (MP4) for an avatar with supplied visemes
• WS /ws — streaming input and realtime viseme output (for low-latency applications)

Example POST /v1/lipsync (curl):

curl -X POST "http://localhost:8080/v1/lipsync" \
  -F "avatar_id=my_avatar" \
  -F "audio=@/path/to/utterance.wav" \
  -H "Accept: application/json"

Typical JSON response:

{
  "audio_url": "/outputs/1234.wav",
  "visemes": [
    {"viseme": "AA", "start": 0.12, "end": 0.18},
    {"viseme": "M", "start": 0.18, "end": 0.32}
  ],
  "duration": 1.45,
  "metadata": {"engine": "zeroshot-v1"}
}

Use the viseme array to drive avatar blendshapes or morph targets at the specified times.

Use Cases

• Game dialogue: Generate per-line viseme timing for NPC dialogue. Integrate timings into your animation timeline or runtime blendshape system to sync mouth shapes without pre-baking for each line.
• Virtual production / Vtubing: Accept live audio from a stream (WebSocket) and apply viseme events to a realtime avatar renderer for live streams or conferences.
• Accessibility & captioning: Use the viseme timings alongside generated captions to improve playback synchronization and lipreading aids.
• Automated content creation: Batch-render videos with synced avatar speech by submitting text batches to /v1/say and then calling /v1/render to create packaged MP4 files.
• Chatbots and voice assistants: Combine TTS and zeroshot lipsync so spoken responses immediately drive avatar mouth animation, reducing integration complexity.

Quick integration example (Node.js)

import fetch from "node-fetch";
import fs from "fs";

const res = await fetch("http://localhost:8080/v1/say", {
  method: "POST",
  body: JSON.stringify({ text: "Hello, welcome to the demo.", avatar_id: "default" }),
  headers: { "Content-Type": "application/json" }
});
const json = await res.json();
// Save audio file and use viseme timings in your engine
console.log(json.visemes);

For more details, check the GitHub repository for configuration options, sample clients, and deployment examples. The MCP server is intended to be a straightforward building block for fast, zeroshot lipsync integration in development pipelines.