Running Ollama locally?

Ollama slow, swapping, or picking the wrong model? We fix it.

A laptop that handled a 7-billion-parameter model fine is grinding through a 70-billion one, the Modelfile system prompt is being ignored half the time, and the team is not sure whether the server sitting under someone's desk is actually fast enough to serve the whole department. We measure the hardware, pick models that fit, and set the service up so it earns its keep.

  • Since 2015Fixing technology
  • FreeEstimates
  • US teamReal engineers

ollama.log

still broken

What it is

Ollama at a glance

Ollama is an open-source tool for running large language models on your own hardware. It wraps llama.cpp and related runtimes, exposes a REST and CLI interface, and uses a Modelfile format borrowed from Dockerfile syntax to describe which base model, system prompt, template and parameters a named model uses. Models are pulled from the Ollama registry or imported from GGUF files, and the service listens on port 11434 for application traffic. It runs on macOS, Linux and Windows, and takes advantage of Apple Silicon unified memory, NVIDIA CUDA and AMD ROCm where they are available.

The point of Ollama is to keep model inference local: data does not leave the machine, there is no per-token billing, and the engineering team controls which model version is running. The flip side is that the hardware choice becomes the whole story. A 7-billion-parameter model at Q4 quantization fits comfortably on a modern laptop GPU; a 70-billion-parameter model at high quality needs a workstation or a server with at least 48 GB of VRAM or a lot of unified memory, and anything larger starts looking like dedicated infrastructure. Most of the problems we see come down to picking a model that does not fit, running without a reverse proxy in front of the API, or trusting throughput numbers that were measured on different hardware.

Type
Local large language model runtime
First released
2023
License
MIT
Maintained by
Ollama, Inc. and the open-source community
Typically used for
Private chat assistants, local coding help, prototyping without cloud bills, air-gapped AI workloads

The honest version

Ollama pros and cons

No technology is all upside. This is where Ollama earns its place and where it tends to cause trouble.

Where Ollama is strong

  • Data never leaves the machineInference happens locally, so prompts and outputs are not sent to a third-party API. For regulated data and on-premise work this is often the whole reason to pick it.
  • A friendly front-end to llama.cppThe CLI, Modelfiles and REST API hide a lot of llama.cpp detail without taking away the performance it offers.
  • One format for many modelsThe Ollama registry covers Llama, Mistral, Mixtral, Qwen, Gemma, Phi, DeepSeek and more. Once Ollama is set up, trying a new model is a single pull command.
  • Modelfiles keep setups reproducibleSystem prompt, template, parameters and base model live in one text file, which makes a team's model configuration easy to version and share.
  • Good enough for coding and draftingOn modest hardware, quantized 7-to-14-billion-parameter models handle code assistance, summarization and drafting workloads adequately for many internal uses.
  • No per-token billOnce the hardware is bought, additional usage is free in cash terms, which changes the cost conversation for workloads that would otherwise run on a paid API.

Where Ollama bites

  • Hardware is the real constraintSmall models on modest GPUs are fine. Large models at good quality need serious VRAM, and no amount of configuration will make a 70-billion model usable on 8 GB.
  • Quality is not comparable to frontier APIsA quantized open model is a long way behind the best hosted models on hard reasoning, long contexts and tool use. Pretending otherwise leads to bad features.
  • The default install has no authenticationOllama binds to the loopback interface by default, but many teams expose it to a LAN or VPN without adding a reverse proxy, API key or tenant separation. That is a security problem waiting to happen.
  • Throughput falls off hard with concurrencyA single machine running local inference handles one or two concurrent requests gracefully and queues the rest. Serving a team of twenty from one GPU is not a plan.
  • Quantization quality is unevenQ4 and lower quantizations save memory but degrade output quality in ways that only show up on harder prompts. The right tradeoff depends on the task.
  • Updates can change model behaviorPulling the latest tag of a model can swap in a new base, a new template or a new default system prompt, which changes answers without a code change on your side.

Sound familiar?

Common Ollama problems we fix

The Ollama problems that bring people to us, and what you have once each one is fixed.

llama runner process has terminated: signal: killed

The runner gets killed mid-generation

The kernel out-of-memory killer ended the process because the model and its context did not fit in available memory. Often the first sign that the model choice is too large for the hardware.

Right-sized model and quantization for the hardware

cudaMalloc failed: out of memory

The GPU cannot hold the model

A model at this quantization needs more VRAM than the card has, so Ollama falls back to CPU for some layers and the throughput collapses to a few tokens per second.

Layers fit on GPU, throughput back to usable

no auth configured

The API is exposed with no authentication

Ollama is listening on a LAN or VPN interface, and anything on that network can send prompts to the model. There is no built-in API key, so a reverse proxy is the usual answer.

API protected, access logged

Modelfile SYSTEM prompt being overridden

The custom system prompt is not being applied

Client code is sending its own system message or a different template, which overrides what the Modelfile set. The model behaves differently depending on which code path called it.

One consistent prompting path, documented

Response: 2.4 tokens/s on CPU

Inference feels unusably slow

A model running on CPU because it did not fit on the GPU, or a laptop running on battery that throttled the GPU, or a large context that pushed the attention cache off the card.

Hardware, model and context all compatible

quantization dropping output quality

Quantized model gives noticeably worse answers

The team saved memory by picking Q3 or Q2, and now the model repeats itself, misses instructions or forgets facts it previously got right.

Quantization chosen for the task, measured on real prompts

bind: address already in use

Ollama will not start because the port is taken

Another Ollama instance, an IDE plugin, or a container that mapped 11434 is still running. The service refuses to start and users see connection errors until the conflict is resolved.

One Ollama per host, port use managed

pull model manifest: 416

A model pull fails partway through

An interrupted download that left a partial blob, a disk running low on space, or a registry rate limit on a free-tier network. Pulls fail with unhelpful HTTP errors.

Pulls reliable, cache in the right place

What we do

Ollama help, from hardware choice to day-to-day running

Call us to pick the right model for your laptop, or to stand up a shared Ollama server for a team.

Hardware and model choice
Matching models to your actual hardware, from Apple Silicon laptops to GPU workstations and multi-GPU servers, with throughput and quality measured on your real prompts.
Modelfiles and prompts
Modelfiles that pin base, template, system prompt and parameters, so every client in the team hits the same configuration rather than their own one-off.
Deployment
Ollama behind a reverse proxy with authentication, logging, rate limiting and tenant separation, running as a proper service rather than someone's background terminal.
Integration
Wiring Ollama into IDE plugins, internal chat apps, RAG pipelines and automation, with fallbacks to a hosted model when the local answer is not good enough.
Cost and comparison
An honest comparison against OpenAI, Anthropic and other hosted options for your workload, with the breakeven point on hardware and the quality gap written down.
Managed Tech
Ongoing care of the Ollama server, with model version control, security updates, and someone watching performance when the team's usage grows past what one machine can handle.

How it works

From first message to fixed

  1. Look Free estimate

    Tell us what's going wrong. We look at it, then tell you plainly what's wrong and what it will cost.

  2. Fix

    We fix the cause, not the symptom. If something is down, getting it back up comes first.

  3. Harden

    Backups, monitoring, updates and access, so the same problem can't quietly come back.

  4. Hand back or stay on

    You get a plain-language write-up. Or we stay on for hosting, upkeep and new work.

Often found together

Running Ollama alongside something else?

We work on the whole stack, so the problem can be wherever it actually is.

Questions

Ollama questions

Something else? Call+1 737 727 7327

What hardware do we need to run Ollama usefully?

For small models up to about 8 billion parameters at Q4 quantization, a laptop with 16 GB of unified memory or a GPU with 8 to 12 GB of VRAM is enough. Models in the 13 to 34 billion range want 24 GB or more. Running 70-billion-parameter models at good quality means a workstation or server with 48 GB or more of VRAM, or a Mac Studio with 96 to 192 GB of unified memory. We will measure your actual prompts on candidate hardware before recommending a purchase.

Is Ollama a good substitute for OpenAI or Anthropic?

For some workloads, yes, and for others, not yet. Internal coding help, drafting, summarization and classification on sensitive data often run well on local models. Hard reasoning, long-context analysis, tool use and tasks that need frontier quality still favor hosted APIs. The right answer for most teams is a mix: local where the quality is enough and the data must stay inside, hosted where quality matters more than privacy.

How do we expose Ollama to a whole team safely?

Not by changing the listen address and leaving the API open. We put a reverse proxy in front, add authentication, log who is calling what, apply rate limits, and separate tenants if more than one team shares the server. Throughput on a single GPU is limited, so we size the hardware against the expected concurrent load rather than the headline tokens per second.

Why does our local model give worse answers than last week?

Usually a pull of a newer tag swapped in a different base model or a different default template, or the context grew long enough to spill off the GPU and change latency, or someone changed the Modelfile system prompt. We pin model versions explicitly, keep a short evaluation set to measure regression, and treat model changes like code changes.

Can you take over an Ollama setup that someone else built?

Yes. We inventory the models in use, which Modelfiles define them, which applications call the server, and how the hardware is actually performing. Then we move the service onto proper infrastructure, add authentication and logging, and leave a short written map of the setup for whoever runs it next.

How much does Ollama help cost?

We quote for free. Tell us the hardware you have or plan to buy, the use case and what is going wrong, and we return a price along with a short recommendation before touching anything.

Free estimate

Ollama setup worrying you?

Tell us what you're running and what's going wrong. A person reads it and comes back with an estimate.

  • No charge to find out what it will take
  • If it's urgent, say so

Replies come from a person. By sending, you authorize us to email you back. No marketing.