PDGrad: Guiding Diffusion Model for Reference-Based Blind Face Restoration with Pivot Direction Gradient Guidance
Reference-based blind face restoration (RefBFR) has gained considerable attention because it utilizes additional reference images to restore facial images in situations where the degradation is caused by unknown factors, making it particularly useful in real-world applications. Recently, guided diff...
Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2024-11-01
|
| Series: | Sensors |
| Subjects: | |
| Online Access: | https://www.mdpi.com/1424-8220/24/22/7112 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1846152470620798976 |
|---|---|
| author | Geon Min Tae Bok Lee Yong Seok Heo |
| author_facet | Geon Min Tae Bok Lee Yong Seok Heo |
| author_sort | Geon Min |
| collection | DOAJ |
| description | Reference-based blind face restoration (RefBFR) has gained considerable attention because it utilizes additional reference images to restore facial images in situations where the degradation is caused by unknown factors, making it particularly useful in real-world applications. Recently, guided diffusion models have demonstrated exceptional performance in this task without requiring training. They achieve this by integrating gradients of the losses where each loss reflects the different desired properties of the additional external images. However, these approaches fail to consider potential conflicts between gradients of multiple losses, which can lead to sub-optimal results. To address this issue, we introduce Pivot Direction Gradient guidance (PDGrad), a novel gradient adjustment method for RefBFR within a guided diffusion framework. To this end, we first define the loss function based on both low-level and high-level features. For loss at each feature level, both the coarsely restored image and the reference image are fully integrated. In cases of conflicting gradients, a pivot gradient is established for each level and other gradients are aligned to it, ensuring that the strengths of both images are maximized. Additionally, if the magnitude of the adjusted gradient exceeds that of the pivot gradient, it is adaptively scaled according to the ratio between the two, placing greater emphasis on the pivot. Extensive experimental results on the CelebRef-HQ dataset show that the proposed PDGrad significantly outperforms competitive approaches both quantitatively and qualitatively. |
| format | Article |
| id | doaj-art-10af5f93f38a4fabb26d1bd1a3c7fc2c |
| institution | Kabale University |
| issn | 1424-8220 |
| language | English |
| publishDate | 2024-11-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Sensors |
| spelling | doaj-art-10af5f93f38a4fabb26d1bd1a3c7fc2c2024-11-26T18:20:45ZengMDPI AGSensors1424-82202024-11-012422711210.3390/s24227112PDGrad: Guiding Diffusion Model for Reference-Based Blind Face Restoration with Pivot Direction Gradient GuidanceGeon Min0Tae Bok Lee1Yong Seok Heo2Department of Artificial Intelligence, Ajou University, Suwon 16499, Republic of KoreaDepartment of Artificial Intelligence, Ajou University, Suwon 16499, Republic of KoreaDepartment of Artificial Intelligence, Ajou University, Suwon 16499, Republic of KoreaReference-based blind face restoration (RefBFR) has gained considerable attention because it utilizes additional reference images to restore facial images in situations where the degradation is caused by unknown factors, making it particularly useful in real-world applications. Recently, guided diffusion models have demonstrated exceptional performance in this task without requiring training. They achieve this by integrating gradients of the losses where each loss reflects the different desired properties of the additional external images. However, these approaches fail to consider potential conflicts between gradients of multiple losses, which can lead to sub-optimal results. To address this issue, we introduce Pivot Direction Gradient guidance (PDGrad), a novel gradient adjustment method for RefBFR within a guided diffusion framework. To this end, we first define the loss function based on both low-level and high-level features. For loss at each feature level, both the coarsely restored image and the reference image are fully integrated. In cases of conflicting gradients, a pivot gradient is established for each level and other gradients are aligned to it, ensuring that the strengths of both images are maximized. Additionally, if the magnitude of the adjusted gradient exceeds that of the pivot gradient, it is adaptively scaled according to the ratio between the two, placing greater emphasis on the pivot. Extensive experimental results on the CelebRef-HQ dataset show that the proposed PDGrad significantly outperforms competitive approaches both quantitatively and qualitatively.https://www.mdpi.com/1424-8220/24/22/7112reference-based blind face restorationclassifier guidance diffusion modelconflicting gradients |
| spellingShingle | Geon Min Tae Bok Lee Yong Seok Heo PDGrad: Guiding Diffusion Model for Reference-Based Blind Face Restoration with Pivot Direction Gradient Guidance Sensors reference-based blind face restoration classifier guidance diffusion model conflicting gradients |
| title | PDGrad: Guiding Diffusion Model for Reference-Based Blind Face Restoration with Pivot Direction Gradient Guidance |
| title_full | PDGrad: Guiding Diffusion Model for Reference-Based Blind Face Restoration with Pivot Direction Gradient Guidance |
| title_fullStr | PDGrad: Guiding Diffusion Model for Reference-Based Blind Face Restoration with Pivot Direction Gradient Guidance |
| title_full_unstemmed | PDGrad: Guiding Diffusion Model for Reference-Based Blind Face Restoration with Pivot Direction Gradient Guidance |
| title_short | PDGrad: Guiding Diffusion Model for Reference-Based Blind Face Restoration with Pivot Direction Gradient Guidance |
| title_sort | pdgrad guiding diffusion model for reference based blind face restoration with pivot direction gradient guidance |
| topic | reference-based blind face restoration classifier guidance diffusion model conflicting gradients |
| url | https://www.mdpi.com/1424-8220/24/22/7112 |
| work_keys_str_mv | AT geonmin pdgradguidingdiffusionmodelforreferencebasedblindfacerestorationwithpivotdirectiongradientguidance AT taeboklee pdgradguidingdiffusionmodelforreferencebasedblindfacerestorationwithpivotdirectiongradientguidance AT yongseokheo pdgradguidingdiffusionmodelforreferencebasedblindfacerestorationwithpivotdirectiongradientguidance |