Efficient Fine-Tuning of Large Language Models via a Low-Rank Gradient Estimator
In this paper, we present a Low-Rank Gradient Estimator (LoGE) to accelerate the finetune-time computation of transformers, especially large language models (LLMs). Unlike Parameter-Efficient Fine-Tuning (PEFT) methods, which primarily aim to minimize the number of fine-tuning parameters, LoGE also...
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2024-12-01
|
| Series: | Applied Sciences |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2076-3417/15/1/82 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1841549427137839104 |
|---|---|
| author | Luoming Zhang Zhenyu Lou Yangwei Ying Cheng Yang Hong Zhou |
| author_facet | Luoming Zhang Zhenyu Lou Yangwei Ying Cheng Yang Hong Zhou |
| author_sort | Luoming Zhang |
| collection | DOAJ |
| description | In this paper, we present a Low-Rank Gradient Estimator (LoGE) to accelerate the finetune-time computation of transformers, especially large language models (LLMs). Unlike Parameter-Efficient Fine-Tuning (PEFT) methods, which primarily aim to minimize the number of fine-tuning parameters, LoGE also significantly reduces the computational load of activation gradient calculations by decomposing pre-trained weights and utilizing low-rank matrices during the backward pass. Our approach includes an effective solution for identifying sensitive and important latent subspaces in large models before training with downstream datasets. As LoGE does not alter the network structure, it can be conveniently integrated into existing models. We validated LoGE’s efficacy through comprehensive experiments across various models on various tasks. For the widely used LLaMA model equipped with LoRA, LoGE achieves up to a 1.3× speedup while maintaining graceful accuracy. |
| format | Article |
| id | doaj-art-1beee20bd63a4602bfe585bffe9f8f12 |
| institution | Kabale University |
| issn | 2076-3417 |
| language | English |
| publishDate | 2024-12-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Applied Sciences |
| spelling | doaj-art-1beee20bd63a4602bfe585bffe9f8f122025-01-10T13:14:22ZengMDPI AGApplied Sciences2076-34172024-12-011518210.3390/app15010082Efficient Fine-Tuning of Large Language Models via a Low-Rank Gradient EstimatorLuoming Zhang0Zhenyu Lou1Yangwei Ying2Cheng Yang3Hong Zhou4The College of Biomedical Engineering and Instrument Science, Zhejiang University, Hangzhou 310027, ChinaThe College of Biomedical Engineering and Instrument Science, Zhejiang University, Hangzhou 310027, ChinaThe College of Biomedical Engineering and Instrument Science, Zhejiang University, Hangzhou 310027, ChinaThe College of Biomedical Engineering and Instrument Science, Zhejiang University, Hangzhou 310027, ChinaThe College of Biomedical Engineering and Instrument Science, Zhejiang University, Hangzhou 310027, ChinaIn this paper, we present a Low-Rank Gradient Estimator (LoGE) to accelerate the finetune-time computation of transformers, especially large language models (LLMs). Unlike Parameter-Efficient Fine-Tuning (PEFT) methods, which primarily aim to minimize the number of fine-tuning parameters, LoGE also significantly reduces the computational load of activation gradient calculations by decomposing pre-trained weights and utilizing low-rank matrices during the backward pass. Our approach includes an effective solution for identifying sensitive and important latent subspaces in large models before training with downstream datasets. As LoGE does not alter the network structure, it can be conveniently integrated into existing models. We validated LoGE’s efficacy through comprehensive experiments across various models on various tasks. For the widely used LLaMA model equipped with LoRA, LoGE achieves up to a 1.3× speedup while maintaining graceful accuracy.https://www.mdpi.com/2076-3417/15/1/82low rankgradient estimatorefficient fine-tuningefficient AI |
| spellingShingle | Luoming Zhang Zhenyu Lou Yangwei Ying Cheng Yang Hong Zhou Efficient Fine-Tuning of Large Language Models via a Low-Rank Gradient Estimator Applied Sciences low rank gradient estimator efficient fine-tuning efficient AI |
| title | Efficient Fine-Tuning of Large Language Models via a Low-Rank Gradient Estimator |
| title_full | Efficient Fine-Tuning of Large Language Models via a Low-Rank Gradient Estimator |
| title_fullStr | Efficient Fine-Tuning of Large Language Models via a Low-Rank Gradient Estimator |
| title_full_unstemmed | Efficient Fine-Tuning of Large Language Models via a Low-Rank Gradient Estimator |
| title_short | Efficient Fine-Tuning of Large Language Models via a Low-Rank Gradient Estimator |
| title_sort | efficient fine tuning of large language models via a low rank gradient estimator |
| topic | low rank gradient estimator efficient fine-tuning efficient AI |
| url | https://www.mdpi.com/2076-3417/15/1/82 |
| work_keys_str_mv | AT luomingzhang efficientfinetuningoflargelanguagemodelsviaalowrankgradientestimator AT zhenyulou efficientfinetuningoflargelanguagemodelsviaalowrankgradientestimator AT yangweiying efficientfinetuningoflargelanguagemodelsviaalowrankgradientestimator AT chengyang efficientfinetuningoflargelanguagemodelsviaalowrankgradientestimator AT hongzhou efficientfinetuningoflargelanguagemodelsviaalowrankgradientestimator |