InternLM/xtuner
A next-generation training engine optimized for ultra-large Mixture-of-Experts (MoE) models, offering superior efficiency and scalability.
Core Features
Detailed Introduction
XTuner V1 is an advanced LLM training engine specifically engineered for the challenges of ultra-large-scale Mixture-of-Experts (MoE) models. It departs from traditional 3D parallel architectures by focusing on mainstream MoE training scenarios, enabling efficient "Dropless Training" for models up to 600B parameters without extensive expert parallelism. The engine also boasts robust long sequence support through advanced memory optimization and DeepSpeed Ulysses integration, maintaining stability despite expert load imbalance. Furthermore, XTuner V1 delivers superior training efficiency, supporting models up to 1T parameters and achieving performance benchmarks that surpass conventional parallel schemes, with a strong emphasis on Ascend NPU optimization.