Efficient Fine-Tuning of Large Language Models via a Low-Rank Gradient Estimator

Efficient Fine-Tuning of Large Language Models via a Low-Rank Gradient Estimator

In this paper, we present a Low-Rank Gradient Estimator (LoGE) to accelerate the finetune-time computation of transformers, especially large language models (LLMs). Unlike Parameter-Efficient Fine-Tuning (PEFT) methods, which primarily aim to minimize the number of fine-tuning parameters, LoGE also...

Full description

Saved in:
Bibliographic Details
Main Authors: Luoming Zhang, Zhenyu Lou, Yangwei Ying, Cheng Yang, Hong Zhou
Format: Article
Language:English
Published: MDPI AG 2024-12-01
Series:Applied Sciences
Subjects:
low rank
gradient estimator
efficient fine-tuning
efficient AI
Online Access:https://www.mdpi.com/2076-3417/15/1/82
Tags: Add Tag
No Tags, Be the first to tag this record!

Similar Items

Similar Items