bitsandbytes-foundation/bitsandbytes

A PyTorch library enabling accessible large language models by dramatically reducing memory consumption through k-bit quantization for both inference and training.

Core Features

8-bit optimizers for memory-efficient training with 32-bit performance.

LLM.int8() for 8-bit inference, halving memory without performance degradation.

QLoRA (4-bit quantization) for memory-saving LLM training without performance compromise.

Provides quantization primitives for 8-bit and 4-bit operations.

GitHub Repo

Quick Start

pip install bitsandbytes

Detailed Introduction

bitsandbytes is a crucial PyTorch library designed to democratize access to large language models by tackling their significant memory footprint. It achieves this through advanced k-bit quantization techniques, including 8-bit optimizers for training, LLM.int8() for efficient inference, and QLoRA for 4-bit training. This allows developers to run and train powerful LLMs on more modest hardware, significantly lowering the barrier to entry while maintaining high performance.