Identify the underlying true model from other models for clinical practice using model performance measures
Abstract Objective To assess whether the outcome generation true model could be identified from other candidate models for clinical practice with current conventional model performance measures considering various simulation scenarios and a CVD risk prediction as exemplar. Study design and setting T...
Saved in:
| Main Author: | |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
BMC
2025-01-01
|
| Series: | BMC Medical Research Methodology |
| Subjects: | |
| Online Access: | https://doi.org/10.1186/s12874-025-02457-w |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1841544531676233728 |
|---|---|
| author | Yan Li |
| author_facet | Yan Li |
| author_sort | Yan Li |
| collection | DOAJ |
| description | Abstract Objective To assess whether the outcome generation true model could be identified from other candidate models for clinical practice with current conventional model performance measures considering various simulation scenarios and a CVD risk prediction as exemplar. Study design and setting Thousands of scenarios of true models were used to simulate clinical data, various candidate models and true models were trained on training datasets and then compared on testing datasets with 25 conventional use model performance measures. This consists of univariate simulation (179.2k simulated datasets and over 1.792 million models), multivariate simulation (728k simulated datasets and over 8.736 million models) and a CVD risk prediction case analysis. Results True models had overall C statistic and 95% range of 0.67 (0.51, 0.96) across all scenarios in univariate simulation, 0.81 (0.54, 0.98) in multivariate simulation, 0.85 (0.82, 0.88) in univariate case analysis and 0.85 (0.82, 0.88) in multivariate case analysis. Measures showed very clear differences between the true model and flip-coin model, little or none differences between the true model and candidate models with extra noises, relatively small differences between the true model and proxy models missing causal predictors. Conclusion The study found the true model is not always identified as the “outperformed” model by current conventional measures for binary outcome, even though such true model is presented in the clinical data. New statistical approaches or measures should be established to identify the casual true model from proxy models, especially for those in proxy models with extra noises and/or missing causal predictors. |
| format | Article |
| id | doaj-art-a05d50f5c2aa4b998b95733ffedb5809 |
| institution | Kabale University |
| issn | 1471-2288 |
| language | English |
| publishDate | 2025-01-01 |
| publisher | BMC |
| record_format | Article |
| series | BMC Medical Research Methodology |
| spelling | doaj-art-a05d50f5c2aa4b998b95733ffedb58092025-01-12T12:28:47ZengBMCBMC Medical Research Methodology1471-22882025-01-0125111210.1186/s12874-025-02457-wIdentify the underlying true model from other models for clinical practice using model performance measuresYan Li0School of Mathematical Sciences, Xiamen UniversityAbstract Objective To assess whether the outcome generation true model could be identified from other candidate models for clinical practice with current conventional model performance measures considering various simulation scenarios and a CVD risk prediction as exemplar. Study design and setting Thousands of scenarios of true models were used to simulate clinical data, various candidate models and true models were trained on training datasets and then compared on testing datasets with 25 conventional use model performance measures. This consists of univariate simulation (179.2k simulated datasets and over 1.792 million models), multivariate simulation (728k simulated datasets and over 8.736 million models) and a CVD risk prediction case analysis. Results True models had overall C statistic and 95% range of 0.67 (0.51, 0.96) across all scenarios in univariate simulation, 0.81 (0.54, 0.98) in multivariate simulation, 0.85 (0.82, 0.88) in univariate case analysis and 0.85 (0.82, 0.88) in multivariate case analysis. Measures showed very clear differences between the true model and flip-coin model, little or none differences between the true model and candidate models with extra noises, relatively small differences between the true model and proxy models missing causal predictors. Conclusion The study found the true model is not always identified as the “outperformed” model by current conventional measures for binary outcome, even though such true model is presented in the clinical data. New statistical approaches or measures should be established to identify the casual true model from proxy models, especially for those in proxy models with extra noises and/or missing causal predictors.https://doi.org/10.1186/s12874-025-02457-wClinical risk prediction modelOutcome generation true modelModel performance measuresCardiovascular disease |
| spellingShingle | Yan Li Identify the underlying true model from other models for clinical practice using model performance measures BMC Medical Research Methodology Clinical risk prediction model Outcome generation true model Model performance measures Cardiovascular disease |
| title | Identify the underlying true model from other models for clinical practice using model performance measures |
| title_full | Identify the underlying true model from other models for clinical practice using model performance measures |
| title_fullStr | Identify the underlying true model from other models for clinical practice using model performance measures |
| title_full_unstemmed | Identify the underlying true model from other models for clinical practice using model performance measures |
| title_short | Identify the underlying true model from other models for clinical practice using model performance measures |
| title_sort | identify the underlying true model from other models for clinical practice using model performance measures |
| topic | Clinical risk prediction model Outcome generation true model Model performance measures Cardiovascular disease |
| url | https://doi.org/10.1186/s12874-025-02457-w |
| work_keys_str_mv | AT yanli identifytheunderlyingtruemodelfromothermodelsforclinicalpracticeusingmodelperformancemeasures |