Toward Accurate Quality Assessment of Machine-Generated Infrared Video Using Fréchet Video Distance
Video generation methods have important implications for the fields of visual control and decision-making. Current research often uses the Fréchet Video Distance (FVD) as an evaluation metric for machine-generated video. However, FVD has not been thoroughly verified on non-visible light s...
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
IEEE
2024-01-01
|
| Series: | IEEE Access |
| Subjects: | |
| Online Access: | https://ieeexplore.ieee.org/document/10663414/ |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1846162947291742208 |
|---|---|
| author | Huaizheng Lu Shiwei Wang Dedong Zhang Bin Huang Erkang Chen Yunfeng Sui |
| author_facet | Huaizheng Lu Shiwei Wang Dedong Zhang Bin Huang Erkang Chen Yunfeng Sui |
| author_sort | Huaizheng Lu |
| collection | DOAJ |
| description | Video generation methods have important implications for the fields of visual control and decision-making. Current research often uses the Fréchet Video Distance (FVD) as an evaluation metric for machine-generated video. However, FVD has not been thoroughly verified on non-visible light sources, especially the widely used infrared light. Therefore, there is an urgent need to use real infrared video data to test the reliability and generalization ability of FVD. Toward that goal, we first collected mainstream infrared video datasets and added various types of noise to synthesize infrared videos of different quality levels. Experiments based on synthetic dataset demonstrate the feasibility of using FVD to assess the quality of infrared video. Next, we trained the Pix2PixGAN network using a dataset containing aligned visible and infrared image pairs. The trained model can generate videos of different quality levels in the infrared light domain. With the generated infrared videos, our experiments show that FVD is able to distinguish the quality differences of different infrared videos. In particular, we found that the lack of labeled infrared dataset and relatively small dataset size of infrared videos has a negative impact on calculating credible FVD values. This is because extracting effective infrared video features remains a difficult problem. Our experimental results suggest that infrared video features can be extracted using large-scale visible light video pre-trained I3D models, and their calculated FVD values are even better than those directly using infrared video pre-trained I3D models. Our study provides a basis for using FVD to evaluate the quality of machine-generated videos under multispectral conditions. |
| format | Article |
| id | doaj-art-fc19082cb26646faae926003eadc8c37 |
| institution | Kabale University |
| issn | 2169-3536 |
| language | English |
| publishDate | 2024-01-01 |
| publisher | IEEE |
| record_format | Article |
| series | IEEE Access |
| spelling | doaj-art-fc19082cb26646faae926003eadc8c372024-11-20T00:01:16ZengIEEEIEEE Access2169-35362024-01-011216883716885210.1109/ACCESS.2024.345340610663414Toward Accurate Quality Assessment of Machine-Generated Infrared Video Using Fréchet Video DistanceHuaizheng Lu0https://orcid.org/0009-0009-0104-6741Shiwei Wang1Dedong Zhang2Bin Huang3https://orcid.org/0000-0002-5043-1445Erkang Chen4https://orcid.org/0000-0003-1577-1732Yunfeng Sui5Department of Computer Science and Technology, School of Computer Engineering, Jimei University, Xiamen, ChinaDepartment of Computer Science and Technology, School of Computer Engineering, Jimei University, Xiamen, ChinaDepartment of Computer Science and Technology, School of Computer Engineering, Jimei University, Xiamen, ChinaDepartment of Computer Science and Technology, School of Computer Engineering, Jimei University, Xiamen, ChinaDepartment of Computer Science and Technology, School of Computer Engineering, Jimei University, Xiamen, ChinaResearch Center, Second Research Institute of CAAC, Chengdu, ChinaVideo generation methods have important implications for the fields of visual control and decision-making. Current research often uses the Fréchet Video Distance (FVD) as an evaluation metric for machine-generated video. However, FVD has not been thoroughly verified on non-visible light sources, especially the widely used infrared light. Therefore, there is an urgent need to use real infrared video data to test the reliability and generalization ability of FVD. Toward that goal, we first collected mainstream infrared video datasets and added various types of noise to synthesize infrared videos of different quality levels. Experiments based on synthetic dataset demonstrate the feasibility of using FVD to assess the quality of infrared video. Next, we trained the Pix2PixGAN network using a dataset containing aligned visible and infrared image pairs. The trained model can generate videos of different quality levels in the infrared light domain. With the generated infrared videos, our experiments show that FVD is able to distinguish the quality differences of different infrared videos. In particular, we found that the lack of labeled infrared dataset and relatively small dataset size of infrared videos has a negative impact on calculating credible FVD values. This is because extracting effective infrared video features remains a difficult problem. Our experimental results suggest that infrared video features can be extracted using large-scale visible light video pre-trained I3D models, and their calculated FVD values are even better than those directly using infrared video pre-trained I3D models. Our study provides a basis for using FVD to evaluate the quality of machine-generated videos under multispectral conditions.https://ieeexplore.ieee.org/document/10663414/Machine-generated infrared videoFréchet video distanceI3D modelcorrelation analysisGAN |
| spellingShingle | Huaizheng Lu Shiwei Wang Dedong Zhang Bin Huang Erkang Chen Yunfeng Sui Toward Accurate Quality Assessment of Machine-Generated Infrared Video Using Fréchet Video Distance IEEE Access Machine-generated infrared video Fréchet video distance I3D model correlation analysis GAN |
| title | Toward Accurate Quality Assessment of Machine-Generated Infrared Video Using Fréchet Video Distance |
| title_full | Toward Accurate Quality Assessment of Machine-Generated Infrared Video Using Fréchet Video Distance |
| title_fullStr | Toward Accurate Quality Assessment of Machine-Generated Infrared Video Using Fréchet Video Distance |
| title_full_unstemmed | Toward Accurate Quality Assessment of Machine-Generated Infrared Video Using Fréchet Video Distance |
| title_short | Toward Accurate Quality Assessment of Machine-Generated Infrared Video Using Fréchet Video Distance |
| title_sort | toward accurate quality assessment of machine generated infrared video using fr x00e9 chet video distance |
| topic | Machine-generated infrared video Fréchet video distance I3D model correlation analysis GAN |
| url | https://ieeexplore.ieee.org/document/10663414/ |
| work_keys_str_mv | AT huaizhenglu towardaccuratequalityassessmentofmachinegeneratedinfraredvideousingfrx00e9chetvideodistance AT shiweiwang towardaccuratequalityassessmentofmachinegeneratedinfraredvideousingfrx00e9chetvideodistance AT dedongzhang towardaccuratequalityassessmentofmachinegeneratedinfraredvideousingfrx00e9chetvideodistance AT binhuang towardaccuratequalityassessmentofmachinegeneratedinfraredvideousingfrx00e9chetvideodistance AT erkangchen towardaccuratequalityassessmentofmachinegeneratedinfraredvideousingfrx00e9chetvideodistance AT yunfengsui towardaccuratequalityassessmentofmachinegeneratedinfraredvideousingfrx00e9chetvideodistance |