Swin Transformer lightweight: an efficient strategy that combines weight sharing, distillation and pruning
Swin Transformer, as a layered visual transformer with shifted windows, has attracted extensive attention in the field of computer vision due to its exceptional modeling capabilities. However, its high computational complexity limits its applicability on devices with constrained computational resour...
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Article |
| Language: | zho |
| Published: |
Beijing Xintong Media Co., Ltd
2024-09-01
|
| Series: | Dianxin kexue |
| Subjects: | |
| Online Access: | http://www.telecomsci.com/zh/article/doi/10.11959/j.issn.1000-0801.2024209/ |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1841528846270070784 |
|---|---|
| author | HAN Bo ZHOU Shun FAN Jianhua WEI Xianglin HU Yongyang ZHU Yanping |
| author_facet | HAN Bo ZHOU Shun FAN Jianhua WEI Xianglin HU Yongyang ZHU Yanping |
| author_sort | HAN Bo |
| collection | DOAJ |
| description | Swin Transformer, as a layered visual transformer with shifted windows, has attracted extensive attention in the field of computer vision due to its exceptional modeling capabilities. However, its high computational complexity limits its applicability on devices with constrained computational resources. To address this issue, a pruning compression method was proposed, integrating weight sharing and distillation. Initially, weight sharing was implemented across layers, and transformation layers were added to introduce weight transformation, thereby enhancing diversity. Subsequently, a parameter dependency mapping graph for the transformation blocks was constructed and analyzed, and a grouping matrix <italic>F</italic> was built to record the dependency relationships among all parameters and identify parameters for simultaneous pruning. Finally, distillation was then employed to restore the model’s performance. Experiments conducted on the ImageNet-Tiny-200 public dataset demonstrate that, with a reduction of 32% in model computational complexity, the proposed method only results in approximately a 3% performance degradation at minimum. It provides a solution for deploying high-performance artificial intelligence models in environments with limited computational resources. |
| format | Article |
| id | doaj-art-549c9e2a4bb049229af89f1090dcd120 |
| institution | Kabale University |
| issn | 1000-0801 |
| language | zho |
| publishDate | 2024-09-01 |
| publisher | Beijing Xintong Media Co., Ltd |
| record_format | Article |
| series | Dianxin kexue |
| spelling | doaj-art-549c9e2a4bb049229af89f1090dcd1202025-01-15T03:34:01ZzhoBeijing Xintong Media Co., LtdDianxin kexue1000-08012024-09-0140667473366270Swin Transformer lightweight: an efficient strategy that combines weight sharing, distillation and pruningHAN BoZHOU ShunFAN JianhuaWEI XianglinHU YongyangZHU YanpingSwin Transformer, as a layered visual transformer with shifted windows, has attracted extensive attention in the field of computer vision due to its exceptional modeling capabilities. However, its high computational complexity limits its applicability on devices with constrained computational resources. To address this issue, a pruning compression method was proposed, integrating weight sharing and distillation. Initially, weight sharing was implemented across layers, and transformation layers were added to introduce weight transformation, thereby enhancing diversity. Subsequently, a parameter dependency mapping graph for the transformation blocks was constructed and analyzed, and a grouping matrix <italic>F</italic> was built to record the dependency relationships among all parameters and identify parameters for simultaneous pruning. Finally, distillation was then employed to restore the model’s performance. Experiments conducted on the ImageNet-Tiny-200 public dataset demonstrate that, with a reduction of 32% in model computational complexity, the proposed method only results in approximately a 3% performance degradation at minimum. It provides a solution for deploying high-performance artificial intelligence models in environments with limited computational resources.http://www.telecomsci.com/zh/article/doi/10.11959/j.issn.1000-0801.2024209/Swin Transformermodel lightweightinference accelerationpruningdistillationweight sharing |
| spellingShingle | HAN Bo ZHOU Shun FAN Jianhua WEI Xianglin HU Yongyang ZHU Yanping Swin Transformer lightweight: an efficient strategy that combines weight sharing, distillation and pruning Dianxin kexue Swin Transformer model lightweight inference acceleration pruning distillation weight sharing |
| title | Swin Transformer lightweight: an efficient strategy that combines weight sharing, distillation and pruning |
| title_full | Swin Transformer lightweight: an efficient strategy that combines weight sharing, distillation and pruning |
| title_fullStr | Swin Transformer lightweight: an efficient strategy that combines weight sharing, distillation and pruning |
| title_full_unstemmed | Swin Transformer lightweight: an efficient strategy that combines weight sharing, distillation and pruning |
| title_short | Swin Transformer lightweight: an efficient strategy that combines weight sharing, distillation and pruning |
| title_sort | swin transformer lightweight an efficient strategy that combines weight sharing distillation and pruning |
| topic | Swin Transformer model lightweight inference acceleration pruning distillation weight sharing |
| url | http://www.telecomsci.com/zh/article/doi/10.11959/j.issn.1000-0801.2024209/ |
| work_keys_str_mv | AT hanbo swintransformerlightweightanefficientstrategythatcombinesweightsharingdistillationandpruning AT zhoushun swintransformerlightweightanefficientstrategythatcombinesweightsharingdistillationandpruning AT fanjianhua swintransformerlightweightanefficientstrategythatcombinesweightsharingdistillationandpruning AT weixianglin swintransformerlightweightanefficientstrategythatcombinesweightsharingdistillationandpruning AT huyongyang swintransformerlightweightanefficientstrategythatcombinesweightsharingdistillationandpruning AT zhuyanping swintransformerlightweightanefficientstrategythatcombinesweightsharingdistillationandpruning |