Inversion-Based Face Swapping With Diffusion Model
Face swapping involves replacing a face in an image with another face and ensuring the seamless integration of the source face into the target image. Previous studies have primarily utilized generative adversarial network-based models for face swapping. This paper introduces inversion-based face swa...
Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
IEEE
2025-01-01
|
| Series: | IEEE Access |
| Subjects: | |
| Online Access: | https://ieeexplore.ieee.org/document/10804772/ |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1841536209884545024 |
|---|---|
| author | Daehyun Yoo Hongchul Lee Jiho Kim |
| author_facet | Daehyun Yoo Hongchul Lee Jiho Kim |
| author_sort | Daehyun Yoo |
| collection | DOAJ |
| description | Face swapping involves replacing a face in an image with another face and ensuring the seamless integration of the source face into the target image. Previous studies have primarily utilized generative adversarial network-based models for face swapping. This paper introduces inversion-based face swapping (InFS), a novel framework employing diffusion inversion. The key contributions of our work include: 1) a facial attribute encoder that consolidates attribute information into a single embedding vector, utilizing the architecture of the pSp encoder, and 2) an enhanced face swapping pipeline that overcomes pose limitations through reenactment preprocessing addressing the challenge of incorrect face swapping at extreme angles. To preserve the target image’s attribute information that may be lost during the diffusion inversion process, we incorporate the extracted information from the facial attribute encoder. This embedding vector serves as a crucial condition in the diffusion inversion process, facilitating the prediction of noisy images. Subsequently, the predicted noisy image undergoes processing using a pretrained ID conditional DDPM for face swapping. Our experimental results show that InFS outperforms state-of-the-art methods in preserving identity, expression, and shape characteristics of target images. Furthermore, the proposed InFS achieved effective face swapping results without requiring additional guidance and reduces inference time by approximately 6.96 seconds compared to previous diffusion-based approaches. |
| format | Article |
| id | doaj-art-d3421e50abf5439fbfe32c9190e6bdf9 |
| institution | Kabale University |
| issn | 2169-3536 |
| language | English |
| publishDate | 2025-01-01 |
| publisher | IEEE |
| record_format | Article |
| series | IEEE Access |
| spelling | doaj-art-d3421e50abf5439fbfe32c9190e6bdf92025-01-15T00:02:53ZengIEEEIEEE Access2169-35362025-01-01136764677410.1109/ACCESS.2024.351916310804772Inversion-Based Face Swapping With Diffusion ModelDaehyun Yoo0https://orcid.org/0009-0002-3362-8438Hongchul Lee1Jiho Kim2https://orcid.org/0000-0003-3733-8702School of Industrial and Management Engineering, Korea University, Seoul, Republic of KoreaSchool of Industrial and Management Engineering, Korea University, Seoul, Republic of KoreaSchool of Industrial and Management Engineering, Korea University, Seoul, Republic of KoreaFace swapping involves replacing a face in an image with another face and ensuring the seamless integration of the source face into the target image. Previous studies have primarily utilized generative adversarial network-based models for face swapping. This paper introduces inversion-based face swapping (InFS), a novel framework employing diffusion inversion. The key contributions of our work include: 1) a facial attribute encoder that consolidates attribute information into a single embedding vector, utilizing the architecture of the pSp encoder, and 2) an enhanced face swapping pipeline that overcomes pose limitations through reenactment preprocessing addressing the challenge of incorrect face swapping at extreme angles. To preserve the target image’s attribute information that may be lost during the diffusion inversion process, we incorporate the extracted information from the facial attribute encoder. This embedding vector serves as a crucial condition in the diffusion inversion process, facilitating the prediction of noisy images. Subsequently, the predicted noisy image undergoes processing using a pretrained ID conditional DDPM for face swapping. Our experimental results show that InFS outperforms state-of-the-art methods in preserving identity, expression, and shape characteristics of target images. Furthermore, the proposed InFS achieved effective face swapping results without requiring additional guidance and reduces inference time by approximately 6.96 seconds compared to previous diffusion-based approaches.https://ieeexplore.ieee.org/document/10804772/Face swappingdiffusion modelinversionattribute encoderimage generation |
| spellingShingle | Daehyun Yoo Hongchul Lee Jiho Kim Inversion-Based Face Swapping With Diffusion Model IEEE Access Face swapping diffusion model inversion attribute encoder image generation |
| title | Inversion-Based Face Swapping With Diffusion Model |
| title_full | Inversion-Based Face Swapping With Diffusion Model |
| title_fullStr | Inversion-Based Face Swapping With Diffusion Model |
| title_full_unstemmed | Inversion-Based Face Swapping With Diffusion Model |
| title_short | Inversion-Based Face Swapping With Diffusion Model |
| title_sort | inversion based face swapping with diffusion model |
| topic | Face swapping diffusion model inversion attribute encoder image generation |
| url | https://ieeexplore.ieee.org/document/10804772/ |
| work_keys_str_mv | AT daehyunyoo inversionbasedfaceswappingwithdiffusionmodel AT hongchullee inversionbasedfaceswappingwithdiffusionmodel AT jihokim inversionbasedfaceswappingwithdiffusionmodel |