Can sparse neural networks finally be fast?
A newly developed software method makes powerful sparse AI models significantly faster and more memory-efficient without requiring specialized hardware.
Large language models (LLMs) are incredibly capable, but they demand massive memory and computing power. One common technique to reduce this cost is "pruning" — setting redundant neural network connections to zero. In theory, these "sparse" models should run faster and use less memory. In practice, however, modern GPUs are heavily optimized for dense, regular computations and struggle to process irregular sparse data. As a result, removing half the numbers in a model rarely cuts the processing time in half.
To solve this hardware bottleneck, we developed MACKO-SpMV. Our method introduces a novel way to store sparse matrices, paired with a GPU kernel optimized specifically for Sparse Matrix-Vector Multiplication (SpMV) - the core operation used when LLMs generate text token by token.
Our results prove that unstructured sparsity can finally be a practical advantage. At 50% sparsity, MACKO is the first method that achieves a meaningful 1.2-1.5× speedup and 1.5× memory reduction over standard dense representations.
The implementation is entirely open-source, allowing researchers and developers to reproduce the results and explore future expansions, including lower numerical precision, small-batch matrix multiplication, and wider GPU architecture support.
Because it successfully answers how to make powerful AI models faster on current hardware, the research was selected for the 2026 International Conference on Machine Learning (ICML) in Seoul, South Korea - widely considered one of the top three AI conferences in the world. Furthermore, it was awarded a Spotlight designation, an honor reserved for only the top few percent of submissions. The research was supported by grants APVV-24-0045, VEGA-1/0140/2, and VEGA-1/0538/2, and part of the compute was provided by Devana and Leonardo HPC clusters.
References
[1] Vladimír Macko and Vladimír Boža. "MACKO: Sparse matrix-vector multiplication for low sparsity." Forty-third International Conference on Machine Learning. 2026. Online version