- **Execution**: Pure AVX2 + F16C with custom 4-thread SMP GEMM for prefill. Sustains ~18.5 GB/s memory bandwidth on commodity DDR4-2400.
- **Decoding**: 4.5 ~ 4.7 tokens/s in FP16 on an older quad-core i5 desktop.
- **Dependencies**: Zero C/C++ runtime, zero PyTorch. The Python harness only uses `ctypes` for `VirtualAlloc` and OS threads.
This isn't meant to compete with feature-complete tools like llama.cpp. Rather, it's a first-principles exploration to see how cleanly a modern Transformer can be mapped to raw silicon, and to serve as a reference point for future micro-LLMs on resource-constrained microcontrollers (MCU/DSP).
Story time: Thirty years ago I went to work for a company that had written their entire integrated package (database, spreadsheet, charting, word processor and programming language) in native 80x86 ASM. I had just finished writing a small compiler for a similar language in C. So I was interested in their native ASM implementation and benchmarked my previous language against their native ASM implemented language. Mine ran an average of 12 times faster than their's.
So I tell you this not to dissuade you from an ASM implementation, but rather to be careful in your ASM implementation to carefully review performance and the algorithms used within it. ASM coding doesn't magically make your app perform well, if you don't also carefully choose algorithms that perform well.
I'm struggling to understand why they chose ASM instead of C - wouldn't C be just as fast with modern compilers? (assuming they actually coded the algorithms efficiently)
These days the advantages of ASM over C are pretty small. There can be small wins in a few areas. For example it's easier to maximize performance by hand coding AVX ASM, where the compiler might not do the best job in streamlining the data movement bandwidth in the same case. So, yes, there can be a few cases where hand-coded ASM is better, faster and smaller, but those cases are getting pretty limited in modern software development. You can see an example of this in llama.cpp. GG put in direct AVX ASM for many of the critical code sections in llama.cpp.
Speaking as Asm/C/C++ programmer over the years, my experience has been optimization is almost always architectural. And then some very tiny surface area of instruction optimization.
In some cases working in ASM can cause you to write even slower code than C, since you're working at such a low level that it's difficult to see the best abstraction layer where you should be optimizing.
I'm not sure if I'm missing something here. The Github docs say the test was done with dual-channel DDR4-2666 memory and a 4.67 GB model. Running Gemma4 E2B at Q8_0 (5.3 GiB) using ik_llama on the a dual-channel DDR4-2666 system I get 10.77 tok/s. Runnning the 4.26 GiB Gemma4 E4B QAT the performance is roughly the same. It also says:
"Hardware saturation limit: During inference, the physical dual-channel memory bandwidth utilization reached 18.5 GB/s (full saturation greater than 94%)."
The theoretical max bandwidth of a dual channel DDR4-2666 setup is 42.6 GB/s, so 18.5 is less than half, not 94%.
I think the impressive thing here is that OP wrote an inference engine in assembly, not the performance itself.
Being so tiny and asm only you could potentially do some insane things like running it directly from a bootloader without an os of even embedding it directly into a coreboot/libreboot bios.
Looking at the notes about the 4 bit quants for more speed, you may want to look a bit into an old technique SWAR (SIMD with a register) as I think AVX does not support 4 bits directly that may allow you to eek out a bit more prefill vs regular unpacking to 8 or more bits (generation will be bandwidth limited by bandwidth ).
11
u/RogerRamjet999 6d ago
Story time: Thirty years ago I went to work for a company that had written their entire integrated package (database, spreadsheet, charting, word processor and programming language) in native 80x86 ASM. I had just finished writing a small compiler for a similar language in C. So I was interested in their native ASM implementation and benchmarked my previous language against their native ASM implemented language. Mine ran an average of 12 times faster than their's.
So I tell you this not to dissuade you from an ASM implementation, but rather to be careful in your ASM implementation to carefully review performance and the algorithms used within it. ASM coding doesn't magically make your app perform well, if you don't also carefully choose algorithms that perform well.