My personal blog

TurboQuant in gguf-runner: roughly half the memory at nearly the same speed

Thu, 26 Mar 2026 00:00:00 +0000

A more technical follow-up

In the previous post I introduced gguf-runner, a small Rust CLI for running GGUF models locally on CPU.

In the follow-up I wrote about vision support, release binaries, and a number of smaller improvements.

This post is about one of the more practical changes: adding TurboQuant as a new KV-cache mode that cuts KV-cache memory sharply without giving up much throughput.

Repo: https://github.com/apimeister/gguf-runner

Why the KV-cache matters so much

For long-context inference, the KV-cache quietly becomes one of the dominant costs.

Building containers on Codeberg

Sun, 15 Mar 2026 00:00:00 +0000

I have been moving more and more of my smaller projects over to Codeberg lately.

For plain git hosting and Forgejo based CI, it is a very pleasant setup. The one area where I had to do a bit more work than expected was container builds.

At first glance this looks like it should be boring:

write a Dockerfile
add a CI step
build the image

But on a hosted Codeberg runner this is exactly the type of thing where the defaults are not quite the defaults you may be used to from GitHub Actions or from your own machine.

gguf-runner updates: vision support, releases, and many small improvements

Sun, 08 Mar 2026 00:00:00 +0000

A quick follow-up

In the previous post I introduced gguf-runner, a small Rust CLI for running GGUF models locally with a focus on:

CPU-only inference
mmap-based model loading
a small, scriptable command line interface

If you haven’t read that one yet, it explains the motivation and the general design of the project.

This post is a follow-up covering some of the more recent additions, most notably vision support, along with a few practical improvements like GitHub release binaries, better documentation, and a number of performance tweaks.

gguf-runner: a minimal GGUF CLI

Tue, 17 Feb 2026 00:00:00 +0000

I’ve been playing with local LLMs again.

Not in the “let’s build a platform” way. More in the “I want a tiny tool I can keep in ~/bin and forget about” way.

So I built gguf-runner: a small Rust CLI to run GGUF models locally, CPU-only, with a focus on low memory overhead and a clean “pipes and scripts” workflow.

Repo: https://github.com/apimeister/gguf-runner

Memory first: GGUF + mmap

The core idea behind gguf-runner is simple:

Letting an LLM Review Our Pull Requests (So You Don’t Have To)

Fri, 08 Aug 2025 00:00:00 +0000

This post is a cross-post from Down to Zero. Canonical source: https://downtozero.cloud/posts/2025/pr-reviews/

We love automation. We use it to power our infrastructure, to scale workloads down to zero, and—increasingly—to shrink the amount of human attention needed to ship high-quality code. One place that still felt stubbornly manual was pull-request reviews. Between Cursor as our IDE, ChatGPT/Codex for prototyping, and gemini-cli for quick checks, our local workflows were fast—but CI still waited for a human.

An epaper picture frame

Sun, 25 Aug 2024 14:00:00 +0000

I was recently on the lookout for a e-paper version of a picture frame that could be battery powered. After an inital short search, and a short excourse to aliexpress (to build my own version) I found something that resembled what I was looking for.

https://framelabs.eu/en/

This site demonstrates nice picture frames with an embedded e-paper display. It is also battery powered and has quite a long battery life.

Digging through the site, I just found one obstacle (that I also found on aliexpress DIY sets). The display only supports 16 greyscale colors. Which seems a little low, but the restriction comes from the epaper display. So I tried it out by ordering one 13inch picture frame.

A Special Purpose HTTP Proxy in Rust

Wed, 10 Jul 2024 14:00:00 +0000

I recently purchased some IoT device based on a pre-built ESP32 firmaware. I wanted to move the server part into the cloud, so I wanted to secure the HTTP calling request with TLS and some form of authentication.

Since the IoT device did not have any TLS or authentication in place, I decided on locally hosting a proxy which transforms HTTP into HTTPS and inject an authnetication in form of an API-KEY.

Getting Started with Kubernetes

Mon, 05 Jun 2023 10:08:54 +0000

The more engines we migrated, the more we reached the limits of our approach.

Deployments grew more and more complex, since we could not check or validate our YAML before bringing it to an actual machine. Sizing Hardware also was an issue, since adding or removing machines still was a manual task.

The deployment was still error prone and relied on bash script execution on the target host. Analyzing issues become more complex.

Scaling BusinessWorks in Kubernetes

Mon, 05 Jun 2023 10:08:54 +0000

Scaling our services in Kubernetes represented a multitude of problems.

As long as our engine contain multiple process starter, the engine can only be manually scaled. Only the developer/maintainer of that engine knows how it behaves if we scale from 1 instance to 2.

Also every Starter in BusinessWorks comes with its own special need in term of parallelism. For Example a Queue receiver can scale horizontally, where as a Topic Consumer cannot. Also Timer based processes need a guarantee that only one instance is running at one point in time.

Server-Timing HTTP header for Axum in Rust

Tue, 02 May 2023 18:13:50 +0000

The “axum-server-timing” project on GitHub provides an Axum layer to inject the Server-Timing HTTP header into responses. This allows developers to measure and report the performance of server-side operations. The project includes examples demonstrating how to use this layer to include timing information, such as duration and descriptions, in HTTP responses.

https://github.com/JensWalter/axum-server-timing

A HTTP Server-Timing Header for axum

Thu, 20 Apr 2023 00:00:00 +0000

Recently I was wondering how I could track the timings of an API I wrote and was looking for ways to implement something to keep track of the individual stages of a request process.

So the overall timing is logged as time in the access log, usually provided by some ingress provider or load balance. But that only shows part of the story, I also wanted to drill down into the timing of subsections of the process.

A Solar Powered GitHub Runner

Sat, 15 Apr 2023 09:08:05 +0000

This is a cross-posting from downtozero.cloud

The more our little project progresses, the more code we produce. And like most of our community these days, we host our code on GitHub.

So during development we discussed the implications of a DownToZero architecture for something like a GitHub process. We quickly identified two types of actions that need to be performed on the infrastructure.

The first is the CI build, which should always provide immediate feedback to the developer. It checks for compliance as well as code integrity and is usually deeply involved in the development process. These jobs are time-sensitive because someone is usually waiting for them.

Reaping some benefits of containerization

Mon, 27 Feb 2023 18:08:54 +0000

We just started the migration and saw some impressive results very early on.

Deployment

We started with some very heavy custom scripting to deploy large composites. These composite container definitions for BW engines, SQL script, EMS definitions and file system deployment. Overall, this process became so complex that even a “simple” composite deployment could easily take 2-3 hours.

By splitting the composite into containers and building our composite structure on top of Docker Compose, deployment times were now down to minutes (up to 5 minutes for container pulls and restarts). This is a huge improvement over existing deployment times.

Containerizing a BusinessWorks Stack

Mon, 27 Feb 2023 14:08:54 +0000

Containerizing BusinessWorks 5 comes with its own set of challenges. Most of these are related to the surrounding ecosystem and the architectures that have evolved around it.

In a classic scenario, a BusinessWorks engine runs within an administrator domain and reports its lifecycle through Hawk. Deployment of an engine is also typically handled by a combination of these tools.

It is important to note that only BusinessWorks provides the interface implementations. The other tools are used to manage/monitor/maintain BW engines and keep them running.

The Classic BusinessWorks Stack

Mon, 27 Feb 2023 12:08:54 +0000

Since I already defined the software stack as a whole, let define how things worked before the migration started.

How Sources and Dependencies are kept.

All the sources were in one big mono-repo. This repo defined engines that followed the definition of backend systems. So for each system (with all its services) we had a source project. Shared functionality was distributed through project libraries so that it could be pulled into multiple engines.

Modernizing a Classic Tibco stack

Mon, 27 Feb 2023 10:08:54 +0000

This will be a series of posts about the steps we took to migrate an existing environment to the cloud.

All the steps we took here took us about 5 years to implement and I will post several articles about the major steps we took.

For now, this is a work in progress and I will update this page as each new article comes out.

Lets define what we mean by Tibco classic stack

The Classic BusinessWorks stack

Lets start with containerization

Migration to Container Orchestration / Kubernetes

KEDA external Scaler for Oracle in Rust

Sun, 22 May 2022 18:13:50 +0000

The project “keda-oracle-scaler” on GitHub is designed to scale KEDA (Kubernetes-based Event Driven Autoscaling) targets based on the results of Oracle database table queries. It allows KEDA to adjust the number of replicas for a Kubernetes deployment depending on the data selected from an Oracle table. This can help in dynamically scaling applications based on real-time database metrics.

https://github.com/apimeister/keda-oracle-scaler

Write a KEDA external Scaler for Oracle in Rust

Sun, 22 May 2022 10:13:50 +0000

Introduction

KEDA (Kubernetes Event-driven Autoscaling) is as the name says a scaling framwork for kubernetes. This frameword can scale based in internal scalers as well as externally defined scalers. Here I wate to write a Rust based scaler, which implements ths gRPS external scaler interface.

Overview

The proto definition gives the following interface.

service ExternalScaler {
    rpc IsActive(ScaledObjectRef) returns (IsActiveResponse) {}
    rpc StreamIsActive(ScaledObjectRef) returns (stream IsActiveResponse) {}
    rpc GetMetricSpec(ScaledObjectRef) returns (GetMetricSpecResponse) {}
    rpc GetMetrics(GetMetricsRequest) returns (GetMetricsResponse) {}
}

For now I am ignoring the option for the streaming service. That leaves me with the isActive and GetMetrics invocations.

How to access Azure Key Vault in Rust

Sat, 13 Feb 2021 10:13:50 +0000

Introduction

Since the Azure SDK for Rust is still not published on crates.io, here is an example how to use the current git repo for accessing an Azure Key Vault.

Cargo.toml

[package]
name = "keyvault-sample"
version = "0.1.0"
edition = "2018"

[dependencies]
azure_key_vault = { git = "https://github.com/Azure/azure-sdk-for-rust.git" }
azure_identity = { git = "https://github.com/Azure/azure-sdk-for-rust.git" }
tokio = { version = "1", features = ["full"] }

main.rs

use azure_identity::token_credentials::ClientSecretCredential;
use azure_identity::token_credentials::TokenCredentialOptions;
use azure_key_vault::KeyVaultClient;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
  //azure credentials
  let client_id = "11111111-1111-1111-1111-111111111111".to_string();
  let client_secret = "super_secret_key".to_string();
  let tenant_id = "11111111-1111-1111-1111-111111111111".to_string();
  //keyvault data
  let keyvault_name = "my-personal-key-vault";
  let secret_name = "sb-connection-string";

  let creds = ClientSecretCredential::new(tenant_id,client_id,client_secret,
    TokenCredentialOptions::default(),
  );
  let mut client = KeyVaultClient::new(&creds, keyvault_name);

  let secret = client.get_secret(secret_name).await?;
  println!("{}",secret.value());
  Ok(())
}

DownToZero.cloud - a sustainable cloud alternative

Tue, 29 Dec 2020 02:13:50 +0000

As the creator of Down to Zero, I designed this product for businesses and organizations deeply conscious of their environmental impact and committed to sustainable practices. Down to Zero offers reliable, scalable, and cost-effective cloud solutions specifically aimed at minimizing carbon footprints. Our key target users include eco-friendly companies, startups focused on sustainability, and enterprises looking to reduce energy consumption in their IT operations. Additionally, businesses seeking secure and customizable cloud infrastructure with dedicated support will find immense value in our services.