Ollama exposes an OpenAI-compatible API on localhost:11434 the moment it’s running. If your app already talks to OpenAI, pointing it at a local model is a base URL change, not a rewrite.

Confirm the server is up

ollama serve

If you installed Ollama as a service, it’s likely already running in the background — check before starting a redundant second instance:

curl http://localhost:11434

A response of Ollama is running means you’re good. Pull a model if you haven’t:

ollama pull llama3.1

Option 1: Ollama’s native API

The native endpoint gives you Ollama-specific features (like structured JSON output and per-request model params) that the OpenAI-compat layer doesn’t always expose cleanly.

curl http://localhost:11434/api/generate -d '{
  "model": "llama3.1",
  "prompt": "Explain what a reverse proxy does in one sentence.",
  "stream": false
}'

Set "stream": false when you want the full response in one JSON object. Leave streaming on (the default) and you get newline-delimited JSON chunks instead — correct for a chat UI, wrong if your code expects one parseable object and gets several instead.

For a chat-style interaction with message history:

curl http://localhost:11434/api/chat -d '{
  "model": "llama3.1",
  "messages": [
    {"role": "user", "content": "What is a wildcard SSL cert?"}
  ]
}'

Option 2: OpenAI-compatible endpoint

If you already have code written against the OpenAI SDK, this is the path of least resistance. Point it at Ollama:

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:11434/v1",
    api_key="ollama",  # required by the SDK, ignored by Ollama
)

response = client.chat.completions.create(
    model="llama3.1",
    messages=[{"role": "user", "content": "What is a wildcard SSL cert?"}]
)

print(response.choices[0].message.content)

api_key has to be a non-empty string or the OpenAI SDK refuses to make the request client-side — Ollama itself doesn’t check it, so any placeholder value works.

Using it from Node

import OpenAI from "openai";

const client = new OpenAI({
  baseURL: "http://localhost:11434/v1",
  apiKey: "ollama",
});

const response = await client.chat.completions.create({
  model: "llama3.1",
  messages: [{ role: "user", content: "What is a wildcard SSL cert?" }],
});

console.log(response.choices[0].message.content);

Streaming responses

For anything user-facing, stream tokens instead of waiting for the full response:

stream = client.chat.completions.create(
    model="llama3.1",
    messages=[{"role": "user", "content": "Write a short changelog entry."}],
    stream=True,
)

for chunk in stream:
    if chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="", flush=True)

Without stream=True, your app waits for the entire generation to finish before showing anything — on a longer response with a local model on modest hardware, that can be several seconds of a blank screen for no visible reason.

Making it reachable beyond localhost

By default, Ollama only listens on 127.0.0.1, which is correct for local development but blocks any other machine or container from reaching it. To expose it on your network (say, a Docker container running your app needs to reach Ollama on the host):

OLLAMA_HOST=0.0.0.0:11434 ollama serve

Do this only inside a network you control. There’s no authentication on Ollama’s API by default — binding it to 0.0.0.0 on a machine with any public exposure means anyone who can reach that port can run inference for free on your hardware, or worse, use it as an entry point if your model loading has any exploitable surface. Put it behind a reverse proxy with auth, or keep it on a private network, if it needs to be reachable from anywhere beyond localhost.

The gotcha with model names

model: "llama3.1" has to exactly match a model you’ve already pulled with ollama pull. Typo the tag (llama3.1:8b versus llama3.1) and you get a 404 with a “model not found” error — Ollama doesn’t fuzzy-match or auto-pull on request. Check what’s actually available with ollama list before wiring it into code.