InternLM/xtuner
XTuner V1 is a next-generation training engine specifically designed for ultra-large-scale Mixture-of-Experts (MoE) models, offering superior efficiency and scalability.
Core Features
Detailed Introduction
XTuner V1 is an advanced LLM training engine engineered for the unique challenges of ultra-large-scale Mixture-of-Experts (MoE) models. Moving beyond traditional 3D parallel architectures, it focuses on mainstream MoE training scenarios, enabling efficient 'Dropless Training' for models up to 600B parameters without complex expert parallelism. It boasts memory-efficient long sequence support, handling 200B MoE models with 64k sequence lengths. XTuner V1 delivers superior training throughput, supporting up to 1T parameters, and is optimized for high-performance hardware like NVIDIA H800 and Ascend A3 Supernode, making it a robust solution for cutting-edge AI research and development.