202 lines
5.2 KiB
Markdown
202 lines
5.2 KiB
Markdown
# Distributed Llama Docker Setup for Raspberry Pi
|
|
|
|
This directory contains Docker configurations to run Distributed Llama on Raspberry Pi devices using containers. There are two variants:
|
|
|
|
1. **Controller** (`Dockerfile.controller`) - Downloads models and runs the API server
|
|
2. **Worker** (`Dockerfile.worker`) - Runs worker nodes that connect to the controller
|
|
|
|
## Quick Start with Docker Compose
|
|
|
|
### 1. Download a Model
|
|
|
|
First, download a model using the controller container:
|
|
|
|
```bash
|
|
# Create a models directory
|
|
mkdir -p models
|
|
|
|
# Download a model (this will take some time)
|
|
docker-compose run --rm controller --download llama3_2_3b_instruct_q40
|
|
```
|
|
|
|
### 2. Start the Distributed Setup
|
|
|
|
```bash
|
|
# Start all services (1 controller + 3 workers)
|
|
docker-compose up
|
|
```
|
|
|
|
The API will be available at `http://localhost:9999`
|
|
|
|
### 3. Test the API
|
|
|
|
```bash
|
|
# List available models
|
|
curl http://localhost:9999/v1/models
|
|
|
|
# Send a chat completion request
|
|
curl -X POST http://localhost:9999/v1/chat/completions \
|
|
-H "Content-Type: application/json" \
|
|
-d '{
|
|
"model": "llama",
|
|
"messages": [{"role": "user", "content": "Hello, how are you?"}],
|
|
"max_tokens": 100
|
|
}'
|
|
```
|
|
|
|
## Manual Docker Usage
|
|
|
|
### Building the Images
|
|
|
|
```bash
|
|
# Build controller image
|
|
docker build -f Dockerfile.controller -t distributed-llama-controller .
|
|
|
|
# Build worker image
|
|
docker build -f Dockerfile.worker -t distributed-llama-worker .
|
|
```
|
|
|
|
### Running the Controller
|
|
|
|
```bash
|
|
# Download a model first
|
|
docker run -v ./models:/app/models distributed-llama-controller --download llama3_2_3b_instruct_q40
|
|
|
|
# Run API server (standalone mode, no workers)
|
|
docker run -p 9999:9999 -v ./models:/app/models distributed-llama-controller \
|
|
--model llama3_2_3b_instruct_q40
|
|
|
|
# Run API server with workers
|
|
docker run -p 9999:9999 -v ./models:/app/models distributed-llama-controller \
|
|
--model llama3_2_3b_instruct_q40 \
|
|
--workers 10.0.0.2:9999 10.0.0.3:9999 10.0.0.4:9999
|
|
```
|
|
|
|
### Running Workers
|
|
|
|
```bash
|
|
# Run a worker on default port 9999
|
|
docker run -p 9999:9999 distributed-llama-worker
|
|
|
|
# Run a worker with custom settings
|
|
docker run -p 9998:9998 distributed-llama-worker --port 9998 --nthreads 2
|
|
```
|
|
|
|
## Available Models
|
|
|
|
You can download any of these models:
|
|
|
|
- `llama3_1_8b_instruct_q40`
|
|
- `llama3_1_405b_instruct_q40` (very large, 56 parts)
|
|
- `llama3_2_1b_instruct_q40`
|
|
- `llama3_2_3b_instruct_q40`
|
|
- `llama3_3_70b_instruct_q40`
|
|
- `deepseek_r1_distill_llama_8b_q40`
|
|
- `qwen3_0.6b_q40`
|
|
- `qwen3_1.7b_q40`
|
|
- `qwen3_8b_q40`
|
|
- `qwen3_14b_q40`
|
|
- `qwen3_30b_a3b_q40`
|
|
|
|
## Configuration Options
|
|
|
|
### Controller Options
|
|
|
|
- `--model <name>`: Model name to use (required)
|
|
- `--port <port>`: API server port (default: 9999)
|
|
- `--nthreads <n>`: Number of threads (default: 4)
|
|
- `--max-seq-len <n>`: Maximum sequence length (default: 4096)
|
|
- `--buffer-float-type <type>`: Buffer float type (default: q80)
|
|
- `--workers <addresses>`: Space-separated worker addresses
|
|
- `--download <model>`: Download a model and exit
|
|
|
|
### Worker Options
|
|
|
|
- `--port <port>`: Worker port (default: 9999)
|
|
- `--nthreads <n>`: Number of threads (default: 4)
|
|
|
|
## Environment Variables (Docker Compose)
|
|
|
|
You can customize the setup using environment variables:
|
|
|
|
```bash
|
|
# Set model and thread counts
|
|
MODEL_NAME=llama3_2_1b_instruct_q40 \
|
|
CONTROLLER_NTHREADS=2 \
|
|
WORKER_NTHREADS=2 \
|
|
docker-compose up
|
|
```
|
|
|
|
Available variables:
|
|
- `MODEL_NAME`: Model to use (default: llama3_2_3b_instruct_q40)
|
|
- `CONTROLLER_NTHREADS`: Controller threads (default: 4)
|
|
- `WORKER_NTHREADS`: Worker threads (default: 4)
|
|
- `MAX_SEQ_LEN`: Maximum sequence length (default: 4096)
|
|
- `BUFFER_FLOAT_TYPE`: Buffer float type (default: q80)
|
|
|
|
## Multi-Device Setup
|
|
|
|
To run across multiple Raspberry Pi devices:
|
|
|
|
### Device 1 (Controller)
|
|
```bash
|
|
# Run controller
|
|
docker run -p 9999:9999 -v ./models:/app/models distributed-llama-controller \
|
|
--model llama3_2_3b_instruct_q40 \
|
|
--workers 10.0.0.2:9999 10.0.0.3:9999 10.0.0.4:9999
|
|
```
|
|
|
|
### Devices 2-4 (Workers)
|
|
```bash
|
|
# Run worker on each device
|
|
docker run -p 9999:9999 distributed-llama-worker --nthreads 4
|
|
```
|
|
|
|
## Performance Tips
|
|
|
|
1. **Thread Count**: Set `--nthreads` to the number of CPU cores on each device
|
|
2. **Memory**: Larger models require more RAM. Monitor usage with `docker stats`
|
|
3. **Network**: Use wired Ethernet connections for better performance between devices
|
|
4. **Storage**: Use fast SD cards (Class 10 or better) or USB 3.0 storage for model files
|
|
|
|
## Troubleshooting
|
|
|
|
### Model Download Issues
|
|
```bash
|
|
# Check if model files exist
|
|
ls -la models/llama3_2_3b_instruct_q40/
|
|
|
|
# Re-download if corrupted
|
|
docker-compose run --rm controller --download llama3_2_3b_instruct_q40
|
|
```
|
|
|
|
### Worker Connection Issues
|
|
```bash
|
|
# Check worker logs
|
|
docker-compose logs worker1
|
|
|
|
# Test network connectivity
|
|
docker exec -it <controller_container> ping 172.20.0.11
|
|
```
|
|
|
|
### Resource Issues
|
|
```bash
|
|
# Monitor resource usage
|
|
docker stats
|
|
|
|
# Reduce thread count if CPU usage is too high
|
|
CONTROLLER_NTHREADS=2 WORKER_NTHREADS=2 docker-compose up
|
|
```
|
|
|
|
## Web Interface
|
|
|
|
You can use the web chat interface at [llama-ui.js.org](https://llama-ui.js.org/):
|
|
|
|
1. Open the website
|
|
2. Go to settings
|
|
3. Set base URL to: `http://your-pi-ip:9999`
|
|
4. Save and start chatting
|
|
|
|
## License
|
|
|
|
This Docker setup follows the same license as the main Distributed Llama project. |