Analysis of Features in a Sliding Threshold of Observation for Numeric Evaluation (STONE) Curve
Abstract We apply idealized scatterplot distributions to the sliding threshold of observation for numeric evaluation (STONE) curve, a new model assessment metric, to examine the relationship between the STONE curve and the underlying point‐spread distribution. The STONE curve is based on the relativ...
Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Wiley
2022-06-01
|
| Series: | Space Weather |
| Subjects: | |
| Online Access: | https://doi.org/10.1029/2022SW003102 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1841536523992825856 |
|---|---|
| author | Michael W. Liemohn Joshua G. Adam Natalia Yu Ganushkina |
| author_facet | Michael W. Liemohn Joshua G. Adam Natalia Yu Ganushkina |
| author_sort | Michael W. Liemohn |
| collection | DOAJ |
| description | Abstract We apply idealized scatterplot distributions to the sliding threshold of observation for numeric evaluation (STONE) curve, a new model assessment metric, to examine the relationship between the STONE curve and the underlying point‐spread distribution. The STONE curve is based on the relative operating characteristic (ROC) curve but is developed to work with a continuous‐valued set of observations, sweeping both the observed and modeled event identification threshold simultaneously. This is particularly useful for model predictions of time series data as is the case for much of terrestrial weather and space weather. The identical sweep of both the model and observational thresholds results in changes to both the modeled and observed event states as the quadrant boundaries shift. The changes in a data‐model pair's event status result in nonmonotonic features to appear in the STONE curve when compared to an ROC curve for the same observational and model data sets. Such features reveal characteristics in the underlying distributions of the data and model values. Many idealized data sets were created with known distributions, connecting certain scatterplot features to distinct STONE curve signatures. A comprehensive suite of feature‐signature combinations is presented, including their relationship to several other metrics. It is shown that nonmonotonic features appear if a local spread is more than 0.2 of the full domain or if a local bias is more than half of the local spread. The example of real‐time plasma sheet electron modeling is used to show the usefulness of this technique, especially in combination with other metrics. |
| format | Article |
| id | doaj-art-e3b52ece80004ae8b59eaaf47181c398 |
| institution | Kabale University |
| issn | 1542-7390 |
| language | English |
| publishDate | 2022-06-01 |
| publisher | Wiley |
| record_format | Article |
| series | Space Weather |
| spelling | doaj-art-e3b52ece80004ae8b59eaaf47181c3982025-01-14T16:27:09ZengWileySpace Weather1542-73902022-06-01206n/an/a10.1029/2022SW003102Analysis of Features in a Sliding Threshold of Observation for Numeric Evaluation (STONE) CurveMichael W. Liemohn0Joshua G. Adam1Natalia Yu Ganushkina2Department of Climate and Space Sciences and Engineering University of Michigan Ann Arbor MI USADepartment of Climate and Space Sciences and Engineering University of Michigan Ann Arbor MI USADepartment of Climate and Space Sciences and Engineering University of Michigan Ann Arbor MI USAAbstract We apply idealized scatterplot distributions to the sliding threshold of observation for numeric evaluation (STONE) curve, a new model assessment metric, to examine the relationship between the STONE curve and the underlying point‐spread distribution. The STONE curve is based on the relative operating characteristic (ROC) curve but is developed to work with a continuous‐valued set of observations, sweeping both the observed and modeled event identification threshold simultaneously. This is particularly useful for model predictions of time series data as is the case for much of terrestrial weather and space weather. The identical sweep of both the model and observational thresholds results in changes to both the modeled and observed event states as the quadrant boundaries shift. The changes in a data‐model pair's event status result in nonmonotonic features to appear in the STONE curve when compared to an ROC curve for the same observational and model data sets. Such features reveal characteristics in the underlying distributions of the data and model values. Many idealized data sets were created with known distributions, connecting certain scatterplot features to distinct STONE curve signatures. A comprehensive suite of feature‐signature combinations is presented, including their relationship to several other metrics. It is shown that nonmonotonic features appear if a local spread is more than 0.2 of the full domain or if a local bias is more than half of the local spread. The example of real‐time plasma sheet electron modeling is used to show the usefulness of this technique, especially in combination with other metrics.https://doi.org/10.1029/2022SW003102ROC curveSTONE curvedata‐model comparisonmodel validationforecastingplasma sheet electrons |
| spellingShingle | Michael W. Liemohn Joshua G. Adam Natalia Yu Ganushkina Analysis of Features in a Sliding Threshold of Observation for Numeric Evaluation (STONE) Curve Space Weather ROC curve STONE curve data‐model comparison model validation forecasting plasma sheet electrons |
| title | Analysis of Features in a Sliding Threshold of Observation for Numeric Evaluation (STONE) Curve |
| title_full | Analysis of Features in a Sliding Threshold of Observation for Numeric Evaluation (STONE) Curve |
| title_fullStr | Analysis of Features in a Sliding Threshold of Observation for Numeric Evaluation (STONE) Curve |
| title_full_unstemmed | Analysis of Features in a Sliding Threshold of Observation for Numeric Evaluation (STONE) Curve |
| title_short | Analysis of Features in a Sliding Threshold of Observation for Numeric Evaluation (STONE) Curve |
| title_sort | analysis of features in a sliding threshold of observation for numeric evaluation stone curve |
| topic | ROC curve STONE curve data‐model comparison model validation forecasting plasma sheet electrons |
| url | https://doi.org/10.1029/2022SW003102 |
| work_keys_str_mv | AT michaelwliemohn analysisoffeaturesinaslidingthresholdofobservationfornumericevaluationstonecurve AT joshuagadam analysisoffeaturesinaslidingthresholdofobservationfornumericevaluationstonecurve AT nataliayuganushkina analysisoffeaturesinaslidingthresholdofobservationfornumericevaluationstonecurve |