Gguf-Runner on My personal blog

gguf-runner as a Rust library

Thu, 06 Aug 2026 00:00:00 +0000

When I first wrote about gguf-runner, it was exactly what the name suggested: a small binary that loaded a GGUF model and wrote the result to the terminal.

That shape worked well.

It gave me a simple way to run a local model from a shell script, and it kept the project focused. Later it gained vision support, release binaries, and faster CPU kernels, but it was still primarily something you started as a separate process.

Shipping CPU-optimized Rust binaries in container images

Sun, 28 Jun 2026 00:00:00 +0000

In an earlier post about gguf-runner I made a point that kept nagging at me afterwards.

The point was this: prebuilt binaries are compiled for a conservative baseline so they run everywhere, and if you build locally with target-cpu=native instead, the compiler can light up the SIMD instructions your specific CPU supports. On an AMD Ryzen 7 PRO 8700GE that was worth about +20.8% tokens/sec for LLM inference. Real, measurable, free.

TurboQuant in gguf-runner: roughly half the memory at nearly the same speed

Thu, 26 Mar 2026 00:00:00 +0000

A more technical follow-up

In the previous post I introduced gguf-runner, a small Rust CLI for running GGUF models locally on CPU.

In the follow-up I wrote about vision support, release binaries, and a number of smaller improvements.

This post is about one of the more practical changes: adding TurboQuant as a new KV-cache mode that cuts KV-cache memory sharply without giving up much throughput.

Repo: https://github.com/apimeister/gguf-runner

Why the KV-cache matters so much

For long-context inference, the KV-cache quietly becomes one of the dominant costs.

gguf-runner updates: vision support, releases, and many small improvements

Sun, 08 Mar 2026 00:00:00 +0000

A quick follow-up

In the previous post I introduced gguf-runner, a small Rust CLI for running GGUF models locally with a focus on:

CPU-only inference
mmap-based model loading
a small, scriptable command line interface

If you haven’t read that one yet, it explains the motivation and the general design of the project.

This post is a follow-up covering some of the more recent additions, most notably vision support, along with a few practical improvements like GitHub release binaries, better documentation, and a number of performance tweaks.

gguf-runner: a minimal GGUF CLI

Tue, 17 Feb 2026 00:00:00 +0000

I’ve been playing with local LLMs again.

Not in the “let’s build a platform” way. More in the “I want a tiny tool I can keep in ~/bin and forget about” way.

So I built gguf-runner: a small Rust CLI to run GGUF models locally, CPU-only, with a focus on low memory overhead and a clean “pipes and scripts” workflow.

Repo: https://github.com/apimeister/gguf-runner

Memory first: GGUF + mmap

The core idea behind gguf-runner is simple: