Spectral convolutional neural network chip for in-sensor edge computing of incoherent natural light
Abstract Optical neural networks are considered next-generation physical implementations of artificial neural networks, but their capabilities are limited by on-chip integration scale and requirement for coherent light sources. This study proposes a spectral convolutional neural network (SCNN) with...
Saved in:
| Main Authors: | , , , , , , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Nature Portfolio
2025-01-01
|
| Series: | Nature Communications |
| Online Access: | https://doi.org/10.1038/s41467-024-55558-3 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1841559285126922240 |
|---|---|
| author | Kaiyu Cui Shijie Rao Sheng Xu Yidong Huang Xusheng Cai Zhilei Huang Yu Wang Xue Feng Fang Liu Wei Zhang Yali Li Shengjin Wang |
| author_facet | Kaiyu Cui Shijie Rao Sheng Xu Yidong Huang Xusheng Cai Zhilei Huang Yu Wang Xue Feng Fang Liu Wei Zhang Yali Li Shengjin Wang |
| author_sort | Kaiyu Cui |
| collection | DOAJ |
| description | Abstract Optical neural networks are considered next-generation physical implementations of artificial neural networks, but their capabilities are limited by on-chip integration scale and requirement for coherent light sources. This study proposes a spectral convolutional neural network (SCNN) with matter meta-imaging. The optical convolutional layer is implemented by integrating very large-scale and pixel-aligned spectral filters on CMOS image sensor. It facilitates highly parallel spectral vector-inner products of incident incoherent natural light i.e., the direct information carrier, which empowers in-sensor optical analog computing at extremely high energy efficiency. To the best of our knowledge, this is the first integrated optical computing utilizing natural light. We employ the same SCNN chip for completely different real-world complex tasks and achieve accuracies of over 96% for pathological diagnosis and almost 100% for face anti-spoofing at video rates. These results indicate a feasible and scalable in-sensor edge computing chip of natural light for various portable terminals. |
| format | Article |
| id | doaj-art-6565ca2094d24890af3dee32c9aeb4c1 |
| institution | Kabale University |
| issn | 2041-1723 |
| language | English |
| publishDate | 2025-01-01 |
| publisher | Nature Portfolio |
| record_format | Article |
| series | Nature Communications |
| spelling | doaj-art-6565ca2094d24890af3dee32c9aeb4c12025-01-05T12:38:32ZengNature PortfolioNature Communications2041-17232025-01-0116111010.1038/s41467-024-55558-3Spectral convolutional neural network chip for in-sensor edge computing of incoherent natural lightKaiyu Cui0Shijie Rao1Sheng Xu2Yidong Huang3Xusheng Cai4Zhilei Huang5Yu Wang6Xue Feng7Fang Liu8Wei Zhang9Yali Li10Shengjin Wang11Department of Electronic Engineering, Tsinghua UniversityDepartment of Electronic Engineering, Tsinghua UniversityDepartment of Electronic Engineering, Tsinghua UniversityDepartment of Electronic Engineering, Tsinghua UniversityBeijing Seetrum Technology Co.Beijing Seetrum Technology Co.Beijing Seetrum Technology Co.Department of Electronic Engineering, Tsinghua UniversityDepartment of Electronic Engineering, Tsinghua UniversityDepartment of Electronic Engineering, Tsinghua UniversityDepartment of Electronic Engineering, Tsinghua UniversityDepartment of Electronic Engineering, Tsinghua UniversityAbstract Optical neural networks are considered next-generation physical implementations of artificial neural networks, but their capabilities are limited by on-chip integration scale and requirement for coherent light sources. This study proposes a spectral convolutional neural network (SCNN) with matter meta-imaging. The optical convolutional layer is implemented by integrating very large-scale and pixel-aligned spectral filters on CMOS image sensor. It facilitates highly parallel spectral vector-inner products of incident incoherent natural light i.e., the direct information carrier, which empowers in-sensor optical analog computing at extremely high energy efficiency. To the best of our knowledge, this is the first integrated optical computing utilizing natural light. We employ the same SCNN chip for completely different real-world complex tasks and achieve accuracies of over 96% for pathological diagnosis and almost 100% for face anti-spoofing at video rates. These results indicate a feasible and scalable in-sensor edge computing chip of natural light for various portable terminals.https://doi.org/10.1038/s41467-024-55558-3 |
| spellingShingle | Kaiyu Cui Shijie Rao Sheng Xu Yidong Huang Xusheng Cai Zhilei Huang Yu Wang Xue Feng Fang Liu Wei Zhang Yali Li Shengjin Wang Spectral convolutional neural network chip for in-sensor edge computing of incoherent natural light Nature Communications |
| title | Spectral convolutional neural network chip for in-sensor edge computing of incoherent natural light |
| title_full | Spectral convolutional neural network chip for in-sensor edge computing of incoherent natural light |
| title_fullStr | Spectral convolutional neural network chip for in-sensor edge computing of incoherent natural light |
| title_full_unstemmed | Spectral convolutional neural network chip for in-sensor edge computing of incoherent natural light |
| title_short | Spectral convolutional neural network chip for in-sensor edge computing of incoherent natural light |
| title_sort | spectral convolutional neural network chip for in sensor edge computing of incoherent natural light |
| url | https://doi.org/10.1038/s41467-024-55558-3 |
| work_keys_str_mv | AT kaiyucui spectralconvolutionalneuralnetworkchipforinsensoredgecomputingofincoherentnaturallight AT shijierao spectralconvolutionalneuralnetworkchipforinsensoredgecomputingofincoherentnaturallight AT shengxu spectralconvolutionalneuralnetworkchipforinsensoredgecomputingofincoherentnaturallight AT yidonghuang spectralconvolutionalneuralnetworkchipforinsensoredgecomputingofincoherentnaturallight AT xushengcai spectralconvolutionalneuralnetworkchipforinsensoredgecomputingofincoherentnaturallight AT zhileihuang spectralconvolutionalneuralnetworkchipforinsensoredgecomputingofincoherentnaturallight AT yuwang spectralconvolutionalneuralnetworkchipforinsensoredgecomputingofincoherentnaturallight AT xuefeng spectralconvolutionalneuralnetworkchipforinsensoredgecomputingofincoherentnaturallight AT fangliu spectralconvolutionalneuralnetworkchipforinsensoredgecomputingofincoherentnaturallight AT weizhang spectralconvolutionalneuralnetworkchipforinsensoredgecomputingofincoherentnaturallight AT yalili spectralconvolutionalneuralnetworkchipforinsensoredgecomputingofincoherentnaturallight AT shengjinwang spectralconvolutionalneuralnetworkchipforinsensoredgecomputingofincoherentnaturallight |