OSS Alternative - Discover Top Open Source Alternatives to Popular Software

liucongg/ChatGLM-Finetuning

A toolkit for finetuning ChatGLM series models (ChatGLM-6B, ChatGLM2-6B, ChatGLM3-6B) using various methods like Freeze, Lora, P-tuning, and full parameter training for downstream NLP tasks.

Core Features

Supports ChatGLM, ChatGLM2, ChatGLM3 models

Offers multiple finetuning methods (Freeze, Lora, P-tuning, Full Parameter)

Supports single and multi-GPU training

Designed for specific downstream tasks (information extraction, generation, classification)

Mitigates catastrophic forgetting with single instruction set finetuning

Quick Start

CUDA_VISIBLE_DEVICES=0 deepspeed --master_port 520 train.py --train_path data/spo_0.json --model_name_or_path ChatGLM-6B/ --per_device_train_batch_size 1 --max_len 1560 --max_src_len 1024 --learning_rate 1e-4 --weight_decay 0.1 --num_train_epochs 2 --gradient_accumulation_steps 4 --warmup_ratio 0.1 --mode glm --train_type freeze --freeze_module_name "layers.27.,layers.26.,layers.25.,layers.24." --seed 1234 --ds_file ds_zero2_no_offload.json --gradient_checkpointing --show_loss_step 10 --output_dir ./output-glm

Detailed Introduction

This project provides a comprehensive framework for fine-tuning the ChatGLM series of large language models (ChatGLM-6B, ChatGLM2-6B, ChatGLM3-6B) on specific downstream tasks. It implements various efficient fine-tuning techniques, including Freeze, Lora, P-tuning, and full parameter training, allowing users to compare their effectiveness. The framework supports both single and multi-GPU training environments and is designed to prevent catastrophic forgetting, making it a valuable tool for researchers and developers looking to adapt large models for specialized NLP applications like information extraction, text generation, and classification.