"Microsoft's BitNet b1.58 2B4T, the first natively trained with 1-bit weights, achieves performance akin to full-precision models while drastically reducing needs for computation and resources."
"This model was trained from scratch on a massive dataset, demonstrating an innovative approach to building LLMs that mitigate the typical precision loss found in quantization."
Microsoft has introduced BitNet b1.58 2B4T, a groundbreaking large language model (LLM) that is natively trained with 1-bit weights instead of using quantization from floating point weights. This new approach not only reduces computational and hardware demands but also matches the performance of full-precision models. The extensive training on a 4 trillion token dataset exemplifies Microsoft's innovative efforts to enhance LLM usability for edge devices and in real-time applications, showing advantages in memory footprint, inference latency, and energy consumption over other models.
Read at InfoQ
Unable to calculate read time
Collection
[
|
...
]