#memory-efficiency

#memory-efficiency

Nowadays, everyone uses cross-platform hybrid desktop apps written in JavaScript, ignoring excessive CPU and RAM usage. You most likely use a hybrid, native-like, cross-platform code editor for day-to-day programming activities. It may work fine on your computer because you've upgraded your hardware, since it may have worked slowly before. If you check the resource usage of your favorite code editor, you'll see not megabytes of RAM, but gigabytes of RAM;

IBM attributes those improved characteristics vs. larger models to its hybrid architecture that combines a small amount of standard transformer-style attention layers with a majority of Mamba layers-more specifically, Mamba-2. With 9 Mamba blocks per 1 Transformer block, Granite gets linear scaling vs. context length for the Mamba parts (vs. quadratic scaling in transformers), plus local contextual dependencies from transformer attention (important for in-context learning or few-shots prompting).

Larger memory page sizes can impact memory efficiency and performance, potentially leading to less efficient memory use if pages are not fully utilized.