gguf-runner started as a small command-line tool. It can now be used as a library inside Rust applications, with model loading, token streaming, vision, tools, and conversation handling kept in-process.
A heavy Rust integration test suite kept getting killed by Codeberg's per-job time limit. Here is how a targeted dev profile, a build-once job, and cargo-nextest partitioning got it green again.
How to map the OCI platform variant to a rustc target-cpu so a Rust binary actually uses AVX2/AVX-512 for LLM and image work inside a container — and how to merge the variants into one multi-arch tag.
Implementing TurboQuant in gguf-runner cuts KV-cache memory roughly in half while staying close to Q8 throughput.
gguf-runner gained vision support, ships GitHub release binaries, and received many usability and performance improvements.
A small Rust CLI to run GGUF models locally: mmap loading, CPU-only inference, and a general-purpose terminal runner that can lean on RAM (and swap) for large models.