ODROID-XU4 LLM Performance & RAG Guide

1. LLM Benchmark Results (Qwen 0.5B Q4_K_M)

Testing executed on the 32-bit ARM legacy board using llama.cpp build 925e11799.

Execution Mode	Thread Config	Prompt Processing (pp512)	Text Generation (tg128)
Unbound CPU	`-t 4`	4.48 t/s	2.42 t/s
Pinned to Cortex-A15 OPTIMAL	`taskset -c 4-7 -t 4`	4.49 t/s	2.55 t/s (+5.4%)

Why 4 Cores Beat 8 Cores

The Exynos 5422 features a big.LITTLE architecture (4× Cortex-A7 + 4× Cortex-A15). Threading matrix multiplication across all 8 cores forces the 4 high-performance A15 cores to wait on the slower A7 cores at layer synchronization boundaries. Pinning execution strictly to cores 4–7 yields maximum tokens-per-second.

2. Production Execution Commands

Optimal Live Chat / Completion

taskset -c 4-7 ./build/bin/llama-completion -m qwen0.5b.gguf -p "The top 3 reasons to use a single-board computer are:" -t 4

Live Web Pipeline (Scrape & Grounded Summary)

taskset -c 4-7 ./build/bin/llama-completion -m qwen0.5b.gguf \
  -p "Summarize what this website is about in 2 sentences based ONLY on this text:\n\n$(curl -sL https://webcentive.com | sed 's/<[^>]*>/ /g' | tr -s ' ' | head -n 30)" \
  -t 4 -no-cnv -n 128

3. Handling Large Books & PDF Collections (RAG)

If your documents exceed what fits in memory (or are in PDF/Word format), you cannot feed them raw into llama-completion. Instead, use Retrieval-Augmented Generation (RAG) to query arbitrary document collections on low-RAM devices:

Option A: Python Pipeline

Install lightweight libraries to chunk PDF/Word files and retrieve relevant snippets:

pip install llama-cpp-python langchain pypdf

Step 1: Convert PDFs or Word files to raw text and slice into ~500-token chunks.
Step 2: Run a lightweight semantic search to select only the top matching chunk for your prompt.
Step 3: Feed only that target chunk to qwen0.5b.gguf to keep memory consumption low.

Option B: Native `llama-retrieval` Tool

llama.cpp includes a built-in vector utility for search:

Vector Embeddings: The built-in llama-retrieval binary converts raw text files into vector embeddings.
Fast Indexing: Queries run locally against pre-calculated index files without requiring full document parsing during inference.