title: “Run Qwen3 and MiniCPM5-1B Locally on Linux with llama.cpp (2026 Guide)”
date: 2026-07-14
categories: [“linux-software”]
tags: [“llama.cpp”, “qwen3”, “minicpm5”, “local-llm”, “ai”, “ollama”, “ubuntu”, “fedora”, “arch”, “gpu-inference”]

Run Qwen3 and MiniCPM5-1B Locally on Linux with llama.cpp (2026 Guide)

In 2026, running a real large language model on your own Linux machine is no
longer exotic. Qwen3-8B from Alibaba runs at a useful speed on an 8 GB
consumer GPU, and MiniCPM5-1B from OpenBMB — released May 26, 2026 —
fits in ~0.5 GB at 4-bit quantization and runs on phones, in browsers,
and on the cheapest Linux box you have lying around.

Both models are Apache 2.0 (commercial use OK), both speak English and
Chinese, and both run natively on llama.cpp — no GPU required for the
smaller models, NVIDIA / AMD / Apple Silicon all supported when you do
have hardware.

This guide covers the full path: install llama.cpp, get Qwen3 running,
get MiniCPM5-1B running, and avoid the gotchas that waste an afternoon.

Why llama.cpp (and not Ollama)?

Ollama is the friendliest way to get started — ollama run qwen3:8b
and you’re chatting in 60 seconds. But llama.cpp is the engine
underneath Ollama, and going direct gives you:

Full control over sampling parameters (top_k, min_p, temperature)
Direct GGUF file access — no Docker, no daemon, no API server
Direct file system inspection of your weights
Tool calling, function calling, JSON grammars with full control
Works on CPU, NVIDIA (CUDA), AMD (ROCm/HIP), Apple Silicon (Metal),
Intel (SYCL), and any Vulkan-compatible GPU

For tinkering, debugging, scripting, and CI pipelines, llama.cpp is the
right level. For just chatting with a model, Ollama is fine.

Install llama.cpp on Linux

The llama.cpp repo moved from ggerganov/llama.cpp to ggml-org/llama.cpp
in 2024-2025. The old URL redirects, but new commits go to the new org.
Use the new one going forward.

Option 1 — Build from source (most flexible)

Bash

# Clone the repo
git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp

# Build with CMake (CPU only)
cmake -B build
cmake --build build --config Release -j$(nproc)

# Binaries appear in build/bin/
ls build/bin/llama-cli build/bin/llama-server

For GPU acceleration, add a backend flag before building:

Bash

# NVIDIA GPUs
cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release -j$(nproc)

# AMD GPUs (ROCm/HIP)
cmake -B build -DGGML_HIP=ON
cmake --build build --config Release -j$(nproc)

# Vulkan (Intel, AMD, NVIDIA fallback)
cmake -B build -DGGML_VULKAN=ON
cmake --build build --config Release -j$(nproc)

# Apple Silicon (Metal)
cmake -B build -DGGML_METAL=ON
cmake --build build --config Release -j$(nproc)

After build, optionally install system-wide:

Bash

sudo cp build/bin/llama-* /usr/local/bin/

Option 2 — Homebrew (Linuxbrew)

If you have Homebrew on Linux:

Bash

brew install llama.cpp

Formula is auto-updated with each llama.cpp release.

Option 3 — Snap (Ubuntu-flavored distros)

Bash

# CPU only
sudo snap install llama-cpp

# NVIDIA GPU
sudo snap install llama-cpp +cuda

# AMD GPU
sudo snap install llama-cpp +hip

# Vulkan (Intel, AMD)
sudo snap install llama-cpp +vulkan

# OpenCL (Adreno, Mali)
sudo snap install llama-cpp +opencl

# Auto-detect
sudo snap set llama-cpp backend=auto

Option 4 — Nix

Bash

# Flake-enabled
nix profile install nixpkgs#llama-cpp

# Traditional
nix-env --file '<nixpkgs>' --install --attr llama-cpp

Verify the install

Bash

llama-cli --version
# Expected: llama.cpp version 6000 or newer (late 2026 numbering)

If you get “command not found”, check $PATH includes /usr/local/bin
or wherever you installed.

Hardware Requirements (2026 reality)

Model	Quantization	RAM needed	GPU VRAM	Notes
MiniCPM5-1B	Q4_K_M	~1 GB	~0.7 GB	Runs on phones, Raspberry Pi 5, anything
Qwen3-0.6B	Q4_K_M	~1.5 GB	~0.8 GB	Tiniest useful chat model
Qwen3-4B	Q4_K_M	~3 GB	~2.5 GB	Sweet spot for low-end laptops
Qwen3-8B	Q4_K_M	~6 GB	~5 GB	Sweet spot for most users
Qwen3-14B	Q4_K_M	~12 GB	~9 GB	RTX 4060 Ti 16 GB and up
Qwen3-32B	Q4_K_M	~22 GB	~19 GB	RTX 4090 24 GB
Qwen3-30B-A3B (MoE)	Q4_K_M	~20 GB	~18 GB	Active params = 3B only at inference
Qwen3-235B-A22B (MoE)	Q4_K_M	~140 GB	~130 GB	Multi-GPU only

CPU-only inference is viable for any model up to ~8B Q4 — expect
5-15 tokens/second on a modern desktop CPU (Ryzen 7 / Core i7). Smaller
models (MiniCPM5-1B, Qwen3-0.6B, Qwen3-1.7B) run at full conversational
speed on CPU.

Run Qwen3 Locally

Step 1 — Download a GGUF build

Qwen3 ships GGUF files on Hugging Face. Pick a size and quantization that
fit your hardware:

Bash

# Example: Qwen3-8B in Q4_K_M (~5 GB on disk, runs on 8 GB GPU)
huggingface-cli download Qwen/Qwen3-8B-GGUF 
    qwen3-8b-q4_k_m.gguf 
    --local-dir ~/models/qwen3-8b

# Example: Qwen3-4B in Q4_K_M (laptop-friendly)
huggingface-cli download Qwen/Qwen3-4B-GGUF 
    qwen3-4b-q4_k_m.gguf 
    --local-dir ~/models/qwen3-4b

You can also grab from the Ollama registry
if you prefer pre-converted files. The model card lists which quantizations
are recommended.

Step 2 — Run the model interactively

Bash

llama-cli 
    -m ~/models/qwen3-8b/qwen3-8b-q4_k_m.gguf 
    -c 8192 
    -n 1024 
    --temp 0.7 
    --top-p 0.9 
    --interactive 
    --color 
    -p "You are a helpful assistant.n"

Flags explained:

-m — path to the GGUF model file
-c — context window size in tokens (default 512, too small for chat)
-n — max tokens to generate per response
--temp — sampling temperature (0.0 = deterministic, 1.0 = creative)
--top-p — nucleus sampling (0.9 is a good default)
--interactive — REPL mode (vs. single-shot)
--color — syntax-highlight prompts vs. responses
-p — system prompt

Step 3 — Or run it as a server

If you want an OpenAI-compatible API (for use with anything that calls
the ChatGPT API — editors, scripts, your own code):

Bash

llama-server 
    -m ~/models/qwen3-8b/qwen3-8b-q4_k_m.gguf 
    -c 8192 
    --host 127.0.0.1 
    --port 8080

Then call it from any client:

Bash

curl http://127.0.0.1:8080/v1/chat/completions 
    -H "Content-Type: application/json" 
    -d '{
      "model": "qwen3-8b",
      "messages": [
        {"role": "user", "content": "Explain llama.cpp in 50 words"}
      ]
    }'

The endpoint is OpenAI-compatible, so any tool that talks to OpenAI
(Continue.dev, Cline, Open WebUI, etc.) can point to it by changing the
base URL to http://127.0.0.1:8080/v1.

Step 4 — Qwen3 thinking mode

Qwen3-2507 and later support a thinking mode where the model
deliberates before answering. To enable:

Bash

llama-cli 
    -m ~/models/qwen3-8b-instruct-q4_k_m.gguf 
    -c 32768 
    --temp 0.6 
    --top-p 0.95 
    --interactive 
    -p "You are Qwen3, a helpful assistant. Use <think></think> blocks for reasoning.n"

The model emits <think>...</think> blocks before its final answer.
Higher context (-c 32768) is recommended because thinking expands the
token usage.

Run MiniCPM5-1B Locally

MiniCPM5-1B is the perfect first local LLM — it fits in less than a
gigabyte at 4-bit, runs on anything, and beats Qwen3-2B on most
benchmarks despite being half the size (per OpenBMB’s published numbers).

Step 1 — Download the GGUF

Bash

huggingface-cli download openbmb/MiniCPM5-1B-GGUF 
    MiniCPM5-1B-Q4_K_M.gguf 
    --local-dir ~/models/minicpm5-1b

(Exact filename depends on the latest HF repo layout — check the
openbmb/MiniCPM5-1B-GGUF
page for current files.)

Step 2 — Run it

Bash

llama-cli 
    -m ~/models/minicpm5-1b/MiniCPM5-1B-Q4_K_M.gguf 
    -c 4096 
    -n 512 
    --temp 0.7 
    --top-p 0.95 
    --interactive 
    -p "You are MiniCPM5-1B, a helpful AI assistant.n"

On a modern laptop, expect 30-60 tokens/second on CPU for this model.
On a Raspberry Pi 5, around 5-10 tokens/second — still readable.

Step 3 — MiniCPM5 thinking mode

Like Qwen3, MiniCPM5-1B supports thinking, controlled by the prompt:

Bash

# Enable thinking
llama-cli 
    -m ~/models/minicpm5-1b/MiniCPM5-1B-Q4_K_M.gguf 
    --interactive 
    -p "You are MiniCPM5-1B. /set thinkn"

# Disable thinking (faster, more direct)
llama-cli 
    -m ~/models/minicpm5-1b/MiniCPM5-1B-Q4_K_M.gguf 
    --interactive 
    -p "You are MiniCPM5-1B. /set nothinkn"

Recommended sampling per the model card:

Think mode: temperature=0.9, top_p=0.95
No-think mode: temperature=0.7, top_p=0.95

Step 4 — Tool calling (with SGLang)

For function/tool calling, SGLang is the recommended backend:

Bash

python -m sglang.launch_server 
    --model-path openbmb/MiniCPM5-1B 
    --port 30000 
    --tool-call-parser minicpm5

MiniCPM5 emits XML-style tool calls; SGLang’s minicpm5 parser converts
them to OpenAI-compatible tool_calls natively.

Running Both Models in Parallel

Want Qwen3 and MiniCPM5-1B available at the same time? Run two
llama-server instances on different ports:

Bash

# Terminal 1: MiniCPM5 on port 8080
llama-server -m ~/models/minicpm5-1b/MiniCPM5-1B-Q4_K_M.gguf 
    --port 8080 -c 4096

# Terminal 2: Qwen3 on port 8081
llama-server -m ~/models/qwen3-8b/qwen3-8b-q4_k_m.gguf 
    --port 8081 -c 8192

Or use Ollama for the simplest multi-model setup:

Bash

# Install Ollama
curl -fsSL https://ollama.com/install.sh | sh

# Pull both
ollama pull minicpm5:1b
ollama pull qwen3:8b

# They share the same OpenAI-compatible API at :11434
# ollama run minicpm5:1b
# ollama run qwen3:8b

Common Issues

“Failed to load model: GGML_ASSERT”

Usually means the GGUF file is corrupted or the wrong architecture. Re-download:

Bash

rm ~/models/qwen3-8b/qwen3-8b-q4_k_m.gguf
huggingface-cli download Qwen/Qwen3-8B-GGUF qwen3-8b-q4_k_m.gguf 
    --local-dir ~/models/qwen3-8b --force-download

Slow first-token latency

First-token latency on a freshly-loaded model is dominated by prompt
evaluation (prefill). For long contexts, use -c only as big as you need:

Bash

# BAD: -c 32768 when you only need 2K context
llama-cli -m model.gguf -c 32768 ...

# GOOD: match context to actual need
llama-cli -m model.gguf -c 4096 ...

Out of memory (CUDA)

Reduce model size or context:

Bash

# Switch from Qwen3-8B Q4 to Qwen3-4B Q4 (half the VRAM)
huggingface-cli download Qwen/Qwen3-4B-GGUF qwen3-4b-q4_k_m.gguf 
    --local-dir ~/models/qwen3-4b

# Or reduce context
llama-cli -m model.gguf -c 4096 -ngl 35 ...

-ngl 35 means “offload 35 layers to GPU” — reduce this number if you
have less VRAM. Set it to 0 to run CPU-only.

“tokenizer.json not found”

The GGUF file should embed the tokenizer, but if you’re loading from a
Hugging Face repo with transformers, you also need the tokenizer files:

Bash

huggingface-cli download Qwen/Qwen3-8B-GGUF 
    --local-dir ~/models/qwen3-8b --include "*.json" "*.txt"

GPU not detected

Verify CUDA / ROCm / Metal is visible:

Bash

# NVIDIA
nvidia-smi

# AMD
rocm-smi

# Apple Silicon
system_profiler SPDisplaysDataType | grep Metal

Then check that llama.cpp was built with the right backend flag
(-DGGML_CUDA=ON, etc.) — rebuild if not.

Performance Tips

Use Q4_K_M quantization for the best size/quality tradeoff.
Q5_K_M is slightly better but ~30% bigger; Q8_0 is near-lossless but
double the size.
Match -ngl to your VRAM — every layer offloaded to GPU is one
less layer on CPU. Use nvidia-smi to find your headroom.
Batch prompts with -np for throughput testing.
Use -fa (flash attention) if your llama.cpp build supports it
(most do by default in 2026).
For Qwen3 thinking mode, set -c 32768 minimum — the reasoning
tokens count toward your context.

Final Thoughts

Qwen3-8B and MiniCPM5-1B are the two models I’d recommend in 2026 for
anyone getting started with local LLMs:

Qwen3-8B — if you have an 8 GB+ NVIDIA GPU (or an M-series Mac)
and want quality close to GPT-3.5
MiniCPM5-1B — if you want something that runs anywhere (phones,
Raspberry Pi, laptops without GPUs) and still gives surprisingly good
answers

Both are Apache 2.0, both speak English + Chinese fluently, both run
on llama.cpp without any custom kernel. The fact that you can download
a 1-billion-parameter model in 0.5 GB and have it answer questions at
30+ tokens/second on a laptop CPU is genuinely amazing compared to
where things stood two years ago.

For more on Qwen3 specifically, see the official Qwen3 GitHub
repo. For MiniCPM5-1B, see the
OpenBMB MiniCPM repo.

Run Qwen3 and MiniCPM5-1B Locally on Linux with llama.cpp (2026 Guide)

title: “Run Qwen3 and MiniCPM5-1B Locally on Linux with llama.cpp (2026 Guide)”
date: 2026-07-14
categories: [“linux-software”]
tags: [“llama.cpp”, “qwen3”, “minicpm5”, “local-llm”, “ai”, “ollama”, “ubuntu”, “fedora”, “arch”, “gpu-inference”]

Run Qwen3 and MiniCPM5-1B Locally on Linux with llama.cpp (2026 Guide)

Why llama.cpp (and not Ollama)?

Install llama.cpp on Linux

Option 1 — Build from source (most flexible)

Option 2 — Homebrew (Linuxbrew)

Option 3 — Snap (Ubuntu-flavored distros)

Option 4 — Nix

Verify the install

Hardware Requirements (2026 reality)

Run Qwen3 Locally

Step 1 — Download a GGUF build

Step 2 — Run the model interactively

Step 3 — Or run it as a server

Step 4 — Qwen3 thinking mode

Run MiniCPM5-1B Locally

Step 1 — Download the GGUF

Step 2 — Run it

Step 3 — MiniCPM5 thinking mode

Step 4 — Tool calling (with SGLang)

Running Both Models in Parallel

Common Issues

“Failed to load model: GGML_ASSERT”

Slow first-token latency

Out of memory (CUDA)

“tokenizer.json not found”

GPU not detected

Performance Tips

Final Thoughts

Sources

Like this:

Author

Comments

Write a Reply or Comment Cancel reply

Run Qwen3 and MiniCPM5-1B Locally on Linux with llama.cpp (2026 Guide)

title: “Run Qwen3 and MiniCPM5-1B Locally on Linux with llama.cpp (2026 Guide)” date: 2026-07-14 categories: [“linux-software”] tags: [“llama.cpp”, “qwen3”, “minicpm5”, “local-llm”, “ai”, “ollama”, “ubuntu”, “fedora”, “arch”, “gpu-inference”]

Run Qwen3 and MiniCPM5-1B Locally on Linux with llama.cpp (2026 Guide)

Why llama.cpp (and not Ollama)?

Install llama.cpp on Linux

Option 1 — Build from source (most flexible)

Option 2 — Homebrew (Linuxbrew)

Option 3 — Snap (Ubuntu-flavored distros)

Option 4 — Nix

Verify the install

Hardware Requirements (2026 reality)

Run Qwen3 Locally

Step 1 — Download a GGUF build

Step 2 — Run the model interactively

Step 3 — Or run it as a server

Step 4 — Qwen3 thinking mode

Run MiniCPM5-1B Locally

Step 1 — Download the GGUF

Step 2 — Run it

Step 3 — MiniCPM5 thinking mode

Step 4 — Tool calling (with SGLang)

Running Both Models in Parallel

Common Issues

“Failed to load model: GGML_ASSERT”

Slow first-token latency

Out of memory (CUDA)

“tokenizer.json not found”

GPU not detected

Performance Tips

Final Thoughts

Sources

Share this:

Like this:

Author

Comments

Write a Reply or Comment Cancel reply

title: “Run Qwen3 and MiniCPM5-1B Locally on Linux with llama.cpp (2026 Guide)”
date: 2026-07-14
categories: [“linux-software”]
tags: [“llama.cpp”, “qwen3”, “minicpm5”, “local-llm”, “ai”, “ollama”, “ubuntu”, “fedora”, “arch”, “gpu-inference”]