-
Does Spatial Information Improve Influenza Forecasting?
Authors:
Gabrielle Thivierge,
Aaron Rumack,
F. William Townes
Abstract:
Seasonal influenza forecasting is critical for public health and individual decision making. We investigate whether the inclusion of data about influenza activity in neighboring states can improve point predictions and distribution forecasting of influenza-like illness (ILI) in each US state using statistical regression models. Using CDC FluView ILI data from 2010-2019, we forecast weekly ILI in e…
▽ More
Seasonal influenza forecasting is critical for public health and individual decision making. We investigate whether the inclusion of data about influenza activity in neighboring states can improve point predictions and distribution forecasting of influenza-like illness (ILI) in each US state using statistical regression models. Using CDC FluView ILI data from 2010-2019, we forecast weekly ILI in each US state with quantile, linear, and Poisson autoregressive models fit using different combinations of ILI data from the target state, neighboring states, and US weighted average. Scoring with root mean squared error and weighted interval score indicated that the variants including neighbors and/or the US average showed slightly higher accuracy than models fit only using lagged ILI in the target state, on average. Additionally, the improvement in performance when including neighbors was similar to the improvement when including the US average instead, suggesting the proximity of the neighboring states is not the driver of the slight increase in accuracy.
△ Less
Submitted 22 August, 2024;
originally announced August 2024.
-
Correcting for heterogeneity in real-time epidemiological indicators
Authors:
Aaron Rumack,
Roni Rosenfeld,
F. William Townes
Abstract:
Auxiliary data sources have become increasingly important in epidemiological surveillance, as they are often available at a finer spatial and temporal resolution, larger coverage, and lower latency than traditional surveillance signals. We describe the problem of spatial and temporal heterogeneity in these signals derived from these data sources, where spatial and/or temporal biases are present. W…
▽ More
Auxiliary data sources have become increasingly important in epidemiological surveillance, as they are often available at a finer spatial and temporal resolution, larger coverage, and lower latency than traditional surveillance signals. We describe the problem of spatial and temporal heterogeneity in these signals derived from these data sources, where spatial and/or temporal biases are present. We present a method to use a ``guiding'' signal to correct for these biases and produce a more reliable signal that can be used for modeling and forecasting. The method assumes that the heterogeneity can be approximated by a low-rank matrix and that the temporal heterogeneity is smooth over time. We also present a hyperparameter selection algorithm to choose the parameters representing the matrix rank and degree of temporal smoothness of the corrections. In the absence of ground truth, we use maps and plots to argue that this method does indeed reduce heterogeneity. Reducing heterogeneity from auxiliary data sources greatly increases their utility in modeling and forecasting epidemics.
△ Less
Submitted 28 September, 2023;
originally announced September 2023.
-
Parity Calibration
Authors:
Youngseog Chung,
Aaron Rumack,
Chirag Gupta
Abstract:
In a sequential regression setting, a decision-maker may be primarily concerned with whether the future observation will increase or decrease compared to the current one, rather than the actual value of the future observation. In this context, we introduce the notion of parity calibration, which captures the goal of calibrated forecasting for the increase-decrease (or "parity") event in a timeseri…
▽ More
In a sequential regression setting, a decision-maker may be primarily concerned with whether the future observation will increase or decrease compared to the current one, rather than the actual value of the future observation. In this context, we introduce the notion of parity calibration, which captures the goal of calibrated forecasting for the increase-decrease (or "parity") event in a timeseries. Parity probabilities can be extracted from a forecasted distribution for the output, but we show that such a strategy leads to theoretical unpredictability and poor practical performance. We then observe that although the original task was regression, parity calibration can be expressed as binary calibration. Drawing on this connection, we use an online binary calibration method to achieve parity calibration. We demonstrate the effectiveness of our approach on real-world case studies in epidemiology, weather forecasting, and model-based control in nuclear fusion.
△ Less
Submitted 7 June, 2023; v1 submitted 29 May, 2023;
originally announced May 2023.
-
Comparing trained and untrained probabilistic ensemble forecasts of COVID-19 cases and deaths in the United States
Authors:
Evan L. Ray,
Logan C. Brooks,
Jacob Bien,
Matthew Biggerstaff,
Nikos I. Bosse,
Johannes Bracher,
Estee Y. Cramer,
Sebastian Funk,
Aaron Gerding,
Michael A. Johansson,
Aaron Rumack,
Yijin Wang,
Martha Zorn,
Ryan J. Tibshirani,
Nicholas G. Reich
Abstract:
The U.S. COVID-19 Forecast Hub aggregates forecasts of the short-term burden of COVID-19 in the United States from many contributing teams. We study methods for building an ensemble that combines forecasts from these teams. These experiments have informed the ensemble methods used by the Hub. To be most useful to policy makers, ensemble forecasts must have stable performance in the presence of two…
▽ More
The U.S. COVID-19 Forecast Hub aggregates forecasts of the short-term burden of COVID-19 in the United States from many contributing teams. We study methods for building an ensemble that combines forecasts from these teams. These experiments have informed the ensemble methods used by the Hub. To be most useful to policy makers, ensemble forecasts must have stable performance in the presence of two key characteristics of the component forecasts: (1) occasional misalignment with the reported data, and (2) instability in the relative performance of component forecasters over time. Our results indicate that in the presence of these challenges, an untrained and robust approach to ensembling using an equally weighted median of all component forecasts is a good choice to support public health decision makers. In settings where some contributing forecasters have a stable record of good performance, trained ensembles that give those forecasters higher weight can also be helpful.
△ Less
Submitted 7 June, 2022; v1 submitted 28 January, 2022;
originally announced January 2022.
-
Recalibrating probabilistic forecasts of epidemics
Authors:
Aaron Rumack,
Ryan J. Tibshirani,
Roni Rosenfeld
Abstract:
Distributional forecasts are important for a wide variety of applications, including forecasting epidemics. Often, forecasts are miscalibrated, or unreliable in assigning uncertainty to future events. We present a recalibration method that can be applied to a black-box forecaster given retrospective forecasts and observations, as well as an extension to make this method more effective in recalibra…
▽ More
Distributional forecasts are important for a wide variety of applications, including forecasting epidemics. Often, forecasts are miscalibrated, or unreliable in assigning uncertainty to future events. We present a recalibration method that can be applied to a black-box forecaster given retrospective forecasts and observations, as well as an extension to make this method more effective in recalibrating epidemic forecasts. This method is guaranteed to improve calibration and log score performance when trained and measured in-sample. We also prove that the increase in expected log score of a recalibrated forecaster is equal to the entropy of the PIT distribution. We apply this recalibration method to the 27 influenza forecasters in the FluSight Network and show that recalibration reliably improves forecast accuracy and calibration. This method is effective, robust, and easy to use as a post-processing tool to improve epidemic forecasts.
△ Less
Submitted 12 December, 2021;
originally announced December 2021.
