Momo AI Hub (GPU Enabled)

This container hosts the AI models (STT, TTS, LLM) for Momo with GPU acceleration.

The runtime HTTP contract consumed by Momo is documented in CONTRACT.md.

Prerequisites

1. Docker installed.
2. NVIDIA Container Toolkit installed:

   sudo apt-get install -y nvidia-container-toolkit
   sudo nvidia-ctk runtime configure --runtime=docker
   sudo systemctl restart docker
   

1. Get a Model (On Your Host Machine)

For your RTX 4070 Super (12GB), you have two great choices:

Option A: High Intelligence (Recommended)

Qwen 2.5 14B Instruct (Q4_K_M)

• Why: "Moonshot-class" reasoning and coding. Smarter than Llama 3 8B.
• Provider: bartowski (Verified working).
• Size: ~9.0 GB. (Leaves ~3GB for Voice).
• Download:

mkdir -p models
wget -O models/qwen2.5-14b-instruct-q4.gguf "https://huggingface.co/bartowski/Qwen2.5-14B-Instruct-GGUF/resolve/main/Qwen2.5-14B-Instruct-Q4_K_M.gguf?download=true"

Option B: Fast & Balanced

Llama 3.1 8B Instruct (Q6_K)

• Why: Extremely fast, very reliable, lower memory usage.
• Size: ~6.6 GB. (Leaves ~5.4GB, very comfortable).
• Download:

mkdir -p models
wget -O models/llama3.1-8b-instruct-q6.gguf "https://huggingface.co/bartowski/Meta-Llama-3.1-8B-Instruct-GGUF/resolve/main/Meta-Llama-3.1-8B-Instruct-Q6_K.gguf?download=true"

2. Download and Run

Make sure to run the model download script to get models (LLM and Voice):

./download_model.sh

Then boot the containers:

docker compose up -d

This also starts a Dozzle instance for the AI Hub machine at http://127.0.0.1:9999 by default.

3. Usage

Container Logs (Dozzle)

The AI Hub machine runs its own Dozzle service and only shows AI Hub containers from that host.

• Default URL: http://127.0.0.1:9999
• Override the published port with AI_HUB_DOZZLE_PORT
• Override the bind address with AI_HUB_DOZZLE_BIND_HOST

If Momo Web and AI Hub run on different machines, each machine keeps its own Dozzle instance and Docker socket. Set AI_HUB_DOZZLE_BIND_HOST=0.0.0.0 only if you intentionally want the AI Hub Dozzle UI exposed on your LAN.

Health Check

curl http://localhost:8080/health

Load the Model

(Replace filename with the one you chose)

curl -X POST http://localhost:8080/admin/load_model \
  -H "Content-Type: application/json" \
  -d '{"filename": "qwen2.5-14b-instruct-q4.gguf", "n_gpu_layers": -1}'

Chat Completion (OpenAI Compatible)

curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "local-model",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'