Skip to Content
Self-hostingLocalAIOpenAI API

LocalAI OpenAI API

LocalAI implements OpenAI-compatible /v1/chat/completions, /v1/embeddings, and related routes on port 8080 by default. Clients swap baseURL — no SDK rewrite.

Hub: LocalAI.

Chat completions

import OpenAI from "openai";
 
const client = new OpenAI({
  baseURL: "http://localhost:8080/v1",
  apiKey: "localai",
});
 
const chat = await client.chat.completions.create({
  model: "llama3", // matches YAML `name`
  messages: [{ role: "user", content: "Explain this regex." }],
  max_tokens: 256,
});

model must match the Setup YAML name field exactly.

Embeddings

const embedding = await client.embeddings.create({
  model: "text-embedding-model",
  input: "chunk to embed",
});

Define embedding models in separate YAML with the appropriate backend.

IDE wiring

ToolConfig
ZedProvider base URL → LocalAI
ClineOpenRouter or custom OpenAI URL
Aideropenai-api-base in config

Same pattern as Ollama/vLLM — one baseURL, many logical models.

Metering

Tag requests:

{
  "inference.gateway": "localai",
  "inference.backend": "llama-cpp",
  "model": "llama3",
  "input_tokens": 890,
  "output_tokens": 120
}

Gateway metrics do not replace backend GPU metering — log both layers.

Last updated on
BackendsIDEs