Unified regularity measures for sample-wise learning and generalization
Abstract Fundamental machine learning theory shows that different samples contribute unequally to both the learning and testing processes. Recent studies on deep neural networks (DNNs) suggest that such sample differences are rooted in the distribution of intrinsic pattern information, namely sample...
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Springer
2024-12-01
|
| Series: | Visual Intelligence |
| Subjects: | |
| Online Access: | https://doi.org/10.1007/s44267-024-00069-4 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1841559023378235392 |
|---|---|
| author | Chi Zhang Meng Yuan Xiaoning Ma Yu Liu Haoang Lu Le Wang Yuanqi Su Yuehu Liu |
| author_facet | Chi Zhang Meng Yuan Xiaoning Ma Yu Liu Haoang Lu Le Wang Yuanqi Su Yuehu Liu |
| author_sort | Chi Zhang |
| collection | DOAJ |
| description | Abstract Fundamental machine learning theory shows that different samples contribute unequally to both the learning and testing processes. Recent studies on deep neural networks (DNNs) suggest that such sample differences are rooted in the distribution of intrinsic pattern information, namely sample regularity. Motivated by recent discoveries in network memorization and generalization, we propose a pair of sample regularity measures with a formulation-consistent representation for both processes. Specifically, the cumulative binary training/generalizing loss (CBTL/CBGL), the cumulative number of correct classifications of the training/test sample within the training phase, is proposed to quantify the stability in the memorization-generalization process, while forgetting/mal-generalizing events (ForEvents/MgEvents), i.e., the misclassification of previously learned or generalized samples, are utilized to represent the uncertainty of sample regularity with respect to optimization dynamics. The effectiveness and robustness of the proposed approaches for mini-batch stochastic gradient descent (SGD) optimization are validated through sample-wise analyses. Further training/test sample selection applications show that the proposed measures, which share the unified computing procedure, could benefit both tasks. |
| format | Article |
| id | doaj-art-a98ba18ac0974090902ab7dff32f9d3d |
| institution | Kabale University |
| issn | 2731-9008 |
| language | English |
| publishDate | 2024-12-01 |
| publisher | Springer |
| record_format | Article |
| series | Visual Intelligence |
| spelling | doaj-art-a98ba18ac0974090902ab7dff32f9d3d2025-01-05T12:50:14ZengSpringerVisual Intelligence2731-90082024-12-012112010.1007/s44267-024-00069-4Unified regularity measures for sample-wise learning and generalizationChi Zhang0Meng Yuan1Xiaoning Ma2Yu Liu3Haoang Lu4Le Wang5Yuanqi Su6Yuehu Liu7Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong UniversityInstitute of Artificial Intelligence and Robotics, Xi’an Jiaotong UniversitySchool of Computer Science and Technology, Xi’an Jiaotong UniversityInstitute of Artificial Intelligence and Robotics, Xi’an Jiaotong UniversitySchool of Computer Science and Technology, Xi’an Jiaotong UniversityInstitute of Artificial Intelligence and Robotics, Xi’an Jiaotong UniversitySchool of Computer Science and Technology, Xi’an Jiaotong UniversityInstitute of Artificial Intelligence and Robotics, Xi’an Jiaotong UniversityAbstract Fundamental machine learning theory shows that different samples contribute unequally to both the learning and testing processes. Recent studies on deep neural networks (DNNs) suggest that such sample differences are rooted in the distribution of intrinsic pattern information, namely sample regularity. Motivated by recent discoveries in network memorization and generalization, we propose a pair of sample regularity measures with a formulation-consistent representation for both processes. Specifically, the cumulative binary training/generalizing loss (CBTL/CBGL), the cumulative number of correct classifications of the training/test sample within the training phase, is proposed to quantify the stability in the memorization-generalization process, while forgetting/mal-generalizing events (ForEvents/MgEvents), i.e., the misclassification of previously learned or generalized samples, are utilized to represent the uncertainty of sample regularity with respect to optimization dynamics. The effectiveness and robustness of the proposed approaches for mini-batch stochastic gradient descent (SGD) optimization are validated through sample-wise analyses. Further training/test sample selection applications show that the proposed measures, which share the unified computing procedure, could benefit both tasks.https://doi.org/10.1007/s44267-024-00069-4Network generalizationMemorization-generalization mechanismSample regularityForgetting eventsSample reweighting |
| spellingShingle | Chi Zhang Meng Yuan Xiaoning Ma Yu Liu Haoang Lu Le Wang Yuanqi Su Yuehu Liu Unified regularity measures for sample-wise learning and generalization Visual Intelligence Network generalization Memorization-generalization mechanism Sample regularity Forgetting events Sample reweighting |
| title | Unified regularity measures for sample-wise learning and generalization |
| title_full | Unified regularity measures for sample-wise learning and generalization |
| title_fullStr | Unified regularity measures for sample-wise learning and generalization |
| title_full_unstemmed | Unified regularity measures for sample-wise learning and generalization |
| title_short | Unified regularity measures for sample-wise learning and generalization |
| title_sort | unified regularity measures for sample wise learning and generalization |
| topic | Network generalization Memorization-generalization mechanism Sample regularity Forgetting events Sample reweighting |
| url | https://doi.org/10.1007/s44267-024-00069-4 |
| work_keys_str_mv | AT chizhang unifiedregularitymeasuresforsamplewiselearningandgeneralization AT mengyuan unifiedregularitymeasuresforsamplewiselearningandgeneralization AT xiaoningma unifiedregularitymeasuresforsamplewiselearningandgeneralization AT yuliu unifiedregularitymeasuresforsamplewiselearningandgeneralization AT haoanglu unifiedregularitymeasuresforsamplewiselearningandgeneralization AT lewang unifiedregularitymeasuresforsamplewiselearningandgeneralization AT yuanqisu unifiedregularitymeasuresforsamplewiselearningandgeneralization AT yuehuliu unifiedregularitymeasuresforsamplewiselearningandgeneralization |