Link a Fast Memory Allocator (jemalloc or tcmalloc)
Standard Linux glibc malloc has overhead when handling small memory allocations across multiple OpenMP threads. Statically linking jemalloc replaces glibc with an allocator designed for thread-local caching.


Not significantly for token generation (tg128). Switching from dynamic linking (.so) to fully static binaries (.a) removes runtime dynamic symbol lookups (PLT/GOT overhead), but that only saves nanoseconds during setup—it does not change the core math throughput.
Token generation is almost purely memory-bandwidth bound. Every single token requires loading the entire 462 MiB model weights from system RAM into the CPU cache once. At 5.67 t/s, your ODROID is moving roughly 2.6 GB/s of model weights over its LPDDR3 memory bus, hitting the real-world limit of the hardware.
However, if you want to optimize static dependencies to squeeze out any remaining efficiency, consider these software adjustments:
1. Link a Fast Memory Allocator (jemalloc or tcmalloc)
Standard Linux glibc malloc has overhead when handling small memory allocations across multiple OpenMP threads. Statically linking jemalloc replaces glibc with an allocator designed for thread-local caching.


sudo apt install -y libjemalloc-dev

# Reconfigure adding jemalloc statically
cmake -B build \
  -DLLAMA_BUILD_SERVER=OFF \
  -DGGML_LTO=ON \
  -DCMAKE_BUILD_TYPE=Release \
  -DCMAKE_EXE_LINKER_FLAGS="-ljemalloc" \
  -DCMAKE_C_FLAGS="-mcpu=cortex-a15 -mfpu=neon-vfpv4 -mfloat-abi=hard -O3 -ffast-math -fno-finite-math-only" \
  -DCMAKE_CXX_FLAGS="-mcpu=cortex-a15 -mfpu=neon-vfpv4 -mfloat-abi=hard -O3 -ffast-math -fno-finite-math-only -Wno-psabi"

cmake --build build --config Release -j2 --target llama-bench




2. Quantize to a Smaller GGUF Format
Because memory bandwidth is your bottleneck, reducing the footprint of the model weights is the single most effective way to make token generation faster:
Quantization Type	Memory Footprint	Relative Memory Bandwidth Speedup
Q4_K_M (Current)	~463 MiB	Baseline (~5.67 t/s)
Q3_K_S	~360 MiB	~1.25x faster (~7.0 t/s estimated)
Q2_K	~290 MiB	~1.5x faster (~8.2 t/s estimated)


3. Disable Overhead at Build Time
Ensure no extra logging, debug, or backend code gets bundled into your static executable:


Bash
-DLLAMA_BUILD_SERVER=OFF
-DGGML_BACKEND_DL=OFF       # Statically link backends instead of loading .so plugins at runtime
-DBUILD_SHARED_LIBS=OFF     # Build strictly static libraries (.a)


Are you open to running a Q3_K or Q2_K quantization of Qwen-0.5B to test if the memory bandwidth scaling holds up?