Running llama.cpp?

llama.cpp broken on Metal or wrong on the GGUF? We sort it out.

llama.cpp is why a laptop or a cheap VPS can run a 13 billion parameter model respectably. The friction shows up in GGUF file mismatches, Metal kernels that refuse to compile against a new macOS release, and quantisation choices that silently hurt quality at the fourth decimal of eval score. We shape the build, the model and the sampler against the box you actually run.

  • Since 2015Fixing technology
  • FreeEstimates
  • US teamReal engineers

llama-cpp-run.log

still failing

What it is

llama.cpp at a glance

llama.cpp is a C/C++ implementation of LLaMA-family inference, written by Georgi Gerganov and maintained by a broad community. It runs quantised models efficiently on CPU, Apple Silicon via Metal, CUDA GPUs, ROCm on AMD, and a growing list of other backends including Vulkan and SYCL.

The GGUF file format packages weights, tokenizer and metadata in a single file that any llama.cpp build can load. The project is the foundation for Ollama, LM Studio, Jan, and much of the local-inference ecosystem, and it is where many new models are quantised and tested before they appear in friendlier tools.

Type
Open-source LLM inference engine for local hardware
First released
2023 public release
License
MIT
Maintained by
Georgi Gerganov and the llama.cpp contributors
Typically used for
Local inference on laptops, servers and edge devices; backing library for Ollama and similar tools

The honest version

llama.cpp pros and cons

No technology is all upside. This is where llama.cpp earns its place and where it tends to cause trouble.

Where llama.cpp is strong

  • Runs on hardware most people already ownA modern laptop with 16GB of RAM handles a 7B or 8B quantised model respectably, and Apple Silicon with the unified memory architecture does particularly well.
  • No service dependencyInference happens entirely on your machine; the data never leaves. For regulated or sensitive workloads this is the deciding feature.
  • Wide backend coverageCPU, Metal, CUDA, ROCm, Vulkan and SYCL are all first-class. One build system covers an unusually wide range of accelerators.
  • Compact GGUF formatA single file carries the weights, tokenizer and metadata. Distributing a model is distributing one file, with reproducible hashes.
  • Fast release cadenceNew model architectures usually land within days of release, which keeps the local-inference ecosystem current with the frontier of open-weight models.

Where llama.cpp bites

  • Quantisation choices hide quality lossQ4_K_M looks fine on an eyeball test and may lose a meaningful chunk of reasoning score. The quality cliff between 4-bit and 5-bit matters for serious work.
  • Build system is a C++ projectCMake, compilers, Metal shader toolchains and CUDA versions have to line up. A fresh macOS update can break the Metal build until the project adapts.
  • Context length costs memory quicklyKV cache for a long context scales linearly, and a 32K window on a laptop can exhaust memory before the first token is generated.
  • Throughput ceilingsA good laptop manages maybe 20 to 40 tokens per second on a 7B model; serving tens of concurrent users is not what the library is built for.
  • Model compatibility is architecture-by-architectureSupport for a new model usually arrives quickly, but gated models or exotic architectures may need a wait or a hand-written converter.
  • Observability is minimalTimings and logs are there; anything beyond that is your own instrumentation. Fine for personal use, needs work for production.

Sound familiar?

Common llama.cpp problems we fix

The llama.cpp problems that bring people to us, and what you have once each one is fixed.

unknown model architecture: 'qwen3'

A new model refuses to load

The GGUF file references an architecture tag the installed llama.cpp build does not know yet. Upgrading to a release that adds support, or compiling from source, is usually the only route.

Model loads on a supported build

ggml_metal_init: shader function not found

The Metal backend fails after a macOS update

Apple ships a toolchain change, and the shipped shaders no longer compile cleanly against the new SDK. A rebuild on the latest llama.cpp release usually resolves it within a few days of the update.

Metal backend running again

tensor has wrong shape

Weights fail to load with a shape mismatch

A GGUF file produced with one tokenizer or vocab size against a binary expecting another. Re-quantising from the original safetensors on a current llama.cpp fixes it rather than patching the file.

Clean GGUF built from source weights

failed to allocate buffer of size 10737418240

Memory allocation fails on a large context

A 10GB KV cache does not fit the machine, or a quantisation that halves the cache size would. The lever list is lower context, smaller model, or a tighter quantisation.

Context fits the hardware

n_ctx 8192 greater than n_ctx_train 4096

Extending context past the trained window degrades quality

Rope scaling makes the model accept longer contexts but the output quality suffers past the training window. Either retrain with extended rope or stay inside the trained range.

Context length matches real capability

CUDA error 2: out of memory

CUDA builds OOM despite the quantisation

A GGUF quantised for CPU may not offload layers efficiently to a small GPU, and the split between CPU and GPU is not right. Tuning the layer offload count against free VRAM fixes it.

Hybrid CPU and GPU offload balanced

llama_decode: failed to decode

Generation fails mid-stream with retval=1

A sampler setting producing non-finite logits, a KV cache corruption, or an input token the tokenizer did not expect. A deterministic seed and a minimal reproduction isolate which one.

Generation completes reliably

What we do

llama.cpp help, from one quantisation choice to a shipped product

A single GGUF that will not load, or a full edge deployment; either way we take a look.

Model selection and quantisation
Choosing the right base model and quantisation level against the hardware you actually ship on, with eval numbers behind the pick.
Build tuning
Compiling with the right flags for your CPU, GPU or Apple Silicon so the throughput ceiling is the hardware, not the binary.
Context length engineering
KV cache sizing, rope scaling and prefix caching set up so long contexts do not blow memory or quality.
Edge deployment
Shipping llama.cpp inside a desktop app, a kiosk or an embedded device, with model files handled and updates planned.
Integration
Hooking the library into Python, Rust, Go or a native app with proper streaming, cancellation and error handling.
Managed Tech
We keep the build current, the model files vetted, and the inference running on whatever hardware the product lives on.

How it works

From first message to fixed

  1. Look Free estimate

    Tell us what's going wrong. We look at it, then tell you plainly what's wrong and what it will cost.

  2. Fix

    We fix the cause, not the symptom. If something is down, getting it back up comes first.

  3. Harden

    Backups, monitoring, updates and access, so the same problem can't quietly come back.

  4. Hand back or stay on

    You get a plain-language write-up. Or we stay on for hosting, upkeep and new work.

Often found together

Running llama.cpp alongside something else?

We work on the whole stack, so the problem can be wherever it actually is.

Questions

llama.cpp questions

Something else? Call+1 737 727 7327

Which quantisation should we use with llama.cpp?

For most 7B to 13B models, Q4_K_M is the sweet spot of quality and memory on a laptop; Q5_K_M is noticeably better where RAM allows. For reasoning-heavy tasks Q6_K or Q8_0 preserve more of the original behaviour, at the cost of memory. Quantising below Q4 is a quality cliff rather than a gentle slope; measure before committing.

Why does our Metal build break after a macOS update?

Apple occasionally changes the Metal shader compiler in a way that invalidates shipped shaders. The fix is a rebuild against a current llama.cpp release that adapts to the new toolchain. Pinning the compiler version and keeping a known-good build archived is the pragmatic workaround for shipping products.

Can llama.cpp serve concurrent users?

Yes with the built-in server, but the design favours throughput per user rather than many parallel sessions. For a handful of internal users it is fine; past that, vLLM or a similar engine with continuous batching is a better fit. The honest cutoff is roughly where request concurrency times prompt length starts to exceed the KV cache budget of one process.

Is llama.cpp a safe foundation for a commercial product?

The MIT license allows it, and the project is maintained actively by a large contributor base. The practical risks are architecture support for a new model landing a week late, and a build system that assumes you keep up with upstream. Both are manageable with a release cadence of your own.

How much does llama.cpp help cost?

Our quote is free. Tell us the target hardware, the model you have in mind and how it needs to behave; the scope and the number come back shortly.

Free estimate

llama.cpp giving you trouble?

Tell us what you're running and what's going wrong. A person reads it and comes back with an estimate.

  • No charge to find out what it will take
  • If it's urgent, say so

Replies come from a person. By sending, you authorize us to email you back. No marketing.