Skip to Content
Self-hostingLocalAISetup

LocalAI setup

Install LocalAI with Docker and define models under models/ as YAML files. Each file’s name becomes the OpenAI model parameter clients pass.

Hub: LocalAI.

Docker run

docker run -p 8080:8080 \
  --mount type=bind,src=$PWD/models,dst=/models \
  --mount type=bind,src=$PWD/config,dst=/config \
  localai/localai:latest

Mount paths:

PathPurpose
/modelsYAML configs + weight files
/configOptional global config

First model (llama.cpp backend)

# models/llama3.yaml
name: llama3
backend: llama-cpp
parameters:
  model: models/llama-3.2-8b-instruct.Q4_K_M.gguf
  context_size: 8192
  f16: true
curl http://localhost:8080/v1/models

Multiple models

Add separate YAML files per capability — route cheap models for classify, larger for codegen:

# models/codellama.yaml
name: codellama
backend: llama-cpp
parameters:
  model: models/codellama-13b.Q5_K_M.gguf
  gpu_layers: 99

Backend selection details: Backends.

Warm-up and ops

IssueFix
Slow first requestModel load — send warm-up request
Model not foundClient model must match YAML name exactly
OOMReduce concurrent backends or dedicated nodes
Last updated on
LocalAIBackends