A new model refuses to load
The GGUF file references an architecture tag the installed llama.cpp build does not know yet. Upgrading to a release that adds support, or compiling from source, is usually the only route.
Model loads on a supported build
Running llama.cpp?
llama.cpp is why a laptop or a cheap VPS can run a 13 billion parameter model respectably. The friction shows up in GGUF file mismatches, Metal kernels that refuse to compile against a new macOS release, and quantisation choices that silently hurt quality at the fourth decimal of eval score. We shape the build, the model and the sampler against the box you actually run.
What it is
llama.cpp is a C/C++ implementation of LLaMA-family inference, written by Georgi Gerganov and maintained by a broad community. It runs quantised models efficiently on CPU, Apple Silicon via Metal, CUDA GPUs, ROCm on AMD, and a growing list of other backends including Vulkan and SYCL.
The GGUF file format packages weights, tokenizer and metadata in a single file that any llama.cpp build can load. The project is the foundation for Ollama, LM Studio, Jan, and much of the local-inference ecosystem, and it is where many new models are quantised and tested before they appear in friendlier tools.
The honest version
No technology is all upside. This is where llama.cpp earns its place and where it tends to cause trouble.
Sound familiar?
The llama.cpp problems that bring people to us, and what you have once each one is fixed.
The GGUF file references an architecture tag the installed llama.cpp build does not know yet. Upgrading to a release that adds support, or compiling from source, is usually the only route.
Model loads on a supported build
Apple ships a toolchain change, and the shipped shaders no longer compile cleanly against the new SDK. A rebuild on the latest llama.cpp release usually resolves it within a few days of the update.
Metal backend running again
A GGUF file produced with one tokenizer or vocab size against a binary expecting another. Re-quantising from the original safetensors on a current llama.cpp fixes it rather than patching the file.
Clean GGUF built from source weights
A 10GB KV cache does not fit the machine, or a quantisation that halves the cache size would. The lever list is lower context, smaller model, or a tighter quantisation.
Context fits the hardware
Rope scaling makes the model accept longer contexts but the output quality suffers past the training window. Either retrain with extended rope or stay inside the trained range.
Context length matches real capability
A GGUF quantised for CPU may not offload layers efficiently to a small GPU, and the split between CPU and GPU is not right. Tuning the layer offload count against free VRAM fixes it.
Hybrid CPU and GPU offload balanced
A sampler setting producing non-finite logits, a KV cache corruption, or an input token the tokenizer did not expect. A deterministic seed and a minimal reproduction isolate which one.
Generation completes reliably
What we do
A single GGUF that will not load, or a full edge deployment; either way we take a look.
How it works
Tell us what's going wrong. We look at it, then tell you plainly what's wrong and what it will cost.
We fix the cause, not the symptom. If something is down, getting it back up comes first.
Backups, monitoring, updates and access, so the same problem can't quietly come back.
You get a plain-language write-up. Or we stay on for hosting, upkeep and new work.
Often found together
We work on the whole stack, so the problem can be wherever it actually is.
For most 7B to 13B models, Q4_K_M is the sweet spot of quality and memory on a laptop; Q5_K_M is noticeably better where RAM allows. For reasoning-heavy tasks Q6_K or Q8_0 preserve more of the original behaviour, at the cost of memory. Quantising below Q4 is a quality cliff rather than a gentle slope; measure before committing.
Apple occasionally changes the Metal shader compiler in a way that invalidates shipped shaders. The fix is a rebuild against a current llama.cpp release that adapts to the new toolchain. Pinning the compiler version and keeping a known-good build archived is the pragmatic workaround for shipping products.
Yes with the built-in server, but the design favours throughput per user rather than many parallel sessions. For a handful of internal users it is fine; past that, vLLM or a similar engine with continuous batching is a better fit. The honest cutoff is roughly where request concurrency times prompt length starts to exceed the KV cache budget of one process.
The MIT license allows it, and the project is maintained actively by a large contributor base. The practical risks are architecture support for a new model landing a week late, and a build system that assumes you keep up with upstream. Both are manageable with a release cadence of your own.
Our quote is free. Tell us the target hardware, the model you have in mind and how it needs to behave; the scope and the number come back shortly.
Free estimate
Tell us what you're running and what's going wrong. A person reads it and comes back with an estimate.
Your message is with a real person, and we'll get back to you soon.
Rather talk now? Call us: