Weighted scoringRules: Emphasizing Particular Outcomes When Evaluating Probabilistic Forecasts
When predicting future events, it is common to issue forecasts that are probabilistic, in the form of probability distributions over the range of possible outcomes. Such forecasts can be evaluated using proper scoring rules. Proper scoring rules condense forecast performance into a single numerical...
Saved in:
| Main Author: | |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Foundation for Open Access Statistics
2024-09-01
|
| Series: | Journal of Statistical Software |
| Online Access: | https://www.jstatsoft.org/index.php/jss/article/view/5028 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1846101689276301312 |
|---|---|
| author | Sam Allen |
| author_facet | Sam Allen |
| author_sort | Sam Allen |
| collection | DOAJ |
| description |
When predicting future events, it is common to issue forecasts that are probabilistic, in the form of probability distributions over the range of possible outcomes. Such forecasts can be evaluated using proper scoring rules. Proper scoring rules condense forecast performance into a single numerical value, allowing competing forecasters to be ranked and compared. To facilitate the use of scoring rules in practical applications, the scoringRules package in R provides popular scoring rules for a wide range of forecast distributions. This paper discusses an extension to the scoringRules package that additionally permits the implementation of popular weighted scoring rules. Weighted scoring rules allow particular outcomes to be targeted during forecast evaluation, recognizing that certain outcomes are often of more interest than others when assessing forecast quality. This introduces the potential for very flexible, user-oriented evaluation of probabilistic forecasts. We discuss the theory underlying weighted scoring rules, and describe how they can readily be implemented in practice using scoringRules. Functionality is available for weighted versions of several popular scoring rules, including the logarithmic score, the continuous ranked probability score, and the energy score. Two case studies are presented to demonstrate this, whereby weighted scoring rules are applied to univariate and multivariate probabilistic forecasts in the fields of meteorology and economics.
|
| format | Article |
| id | doaj-art-40d221ece6804ffdb05a58f0c097e715 |
| institution | Kabale University |
| issn | 1548-7660 |
| language | English |
| publishDate | 2024-09-01 |
| publisher | Foundation for Open Access Statistics |
| record_format | Article |
| series | Journal of Statistical Software |
| spelling | doaj-art-40d221ece6804ffdb05a58f0c097e7152024-12-29T00:12:41ZengFoundation for Open Access StatisticsJournal of Statistical Software1548-76602024-09-01110110.18637/jss.v110.i08Weighted scoringRules: Emphasizing Particular Outcomes When Evaluating Probabilistic ForecastsSam Allen0ETH Zürich When predicting future events, it is common to issue forecasts that are probabilistic, in the form of probability distributions over the range of possible outcomes. Such forecasts can be evaluated using proper scoring rules. Proper scoring rules condense forecast performance into a single numerical value, allowing competing forecasters to be ranked and compared. To facilitate the use of scoring rules in practical applications, the scoringRules package in R provides popular scoring rules for a wide range of forecast distributions. This paper discusses an extension to the scoringRules package that additionally permits the implementation of popular weighted scoring rules. Weighted scoring rules allow particular outcomes to be targeted during forecast evaluation, recognizing that certain outcomes are often of more interest than others when assessing forecast quality. This introduces the potential for very flexible, user-oriented evaluation of probabilistic forecasts. We discuss the theory underlying weighted scoring rules, and describe how they can readily be implemented in practice using scoringRules. Functionality is available for weighted versions of several popular scoring rules, including the logarithmic score, the continuous ranked probability score, and the energy score. Two case studies are presented to demonstrate this, whereby weighted scoring rules are applied to univariate and multivariate probabilistic forecasts in the fields of meteorology and economics. https://www.jstatsoft.org/index.php/jss/article/view/5028 |
| spellingShingle | Sam Allen Weighted scoringRules: Emphasizing Particular Outcomes When Evaluating Probabilistic Forecasts Journal of Statistical Software |
| title | Weighted scoringRules: Emphasizing Particular Outcomes When Evaluating Probabilistic Forecasts |
| title_full | Weighted scoringRules: Emphasizing Particular Outcomes When Evaluating Probabilistic Forecasts |
| title_fullStr | Weighted scoringRules: Emphasizing Particular Outcomes When Evaluating Probabilistic Forecasts |
| title_full_unstemmed | Weighted scoringRules: Emphasizing Particular Outcomes When Evaluating Probabilistic Forecasts |
| title_short | Weighted scoringRules: Emphasizing Particular Outcomes When Evaluating Probabilistic Forecasts |
| title_sort | weighted scoringrules emphasizing particular outcomes when evaluating probabilistic forecasts |
| url | https://www.jstatsoft.org/index.php/jss/article/view/5028 |
| work_keys_str_mv | AT samallen weightedscoringrulesemphasizingparticularoutcomeswhenevaluatingprobabilisticforecasts |