lyogavin/airllm
AirLLM optimizes large language model inference memory, enabling 70B LLMs on a single 4GB GPU without quantization, and 405B Llama3.1 on 8GB VRAM.
Core Features
Quick Start
pip install airllmDetailed Introduction
AirLLM is an innovative library designed to democratize access to large language models by significantly optimizing their inference memory footprint. It addresses the critical challenge of running massive models, such as 70B and even 405B Llama3.1, on consumer-grade GPUs with limited VRAM (e.g., 4GB or 8GB), without resorting to traditional compromises like quantization, distillation, or pruning. By enabling efficient on-device inference, AirLLM lowers the barrier to entry for developers and researchers, making powerful LLMs more accessible for local deployment, edge computing, and cost-effective experimentation across various platforms, including MacOS and CPU environments.