-
Flusion: Integrating multiple data sources for accurate influenza predictions
Authors:
Evan L. Ray,
Yijin Wang,
Russell D. Wolfinger,
Nicholas G. Reich
Abstract:
Over the last ten years, the US Centers for Disease Control and Prevention (CDC) has organized an annual influenza forecasting challenge with the motivation that accurate probabilistic forecasts could improve situational awareness and yield more effective public health actions. Starting with the 2021/22 influenza season, the forecasting targets for this challenge have been based on hospital admiss…
▽ More
Over the last ten years, the US Centers for Disease Control and Prevention (CDC) has organized an annual influenza forecasting challenge with the motivation that accurate probabilistic forecasts could improve situational awareness and yield more effective public health actions. Starting with the 2021/22 influenza season, the forecasting targets for this challenge have been based on hospital admissions reported in the CDC's National Healthcare Safety Network (NHSN) surveillance system. Reporting of influenza hospital admissions through NHSN began within the last few years, and as such only a limited amount of historical data are available for this signal. To produce forecasts in the presence of limited data for the target surveillance system, we augmented these data with two signals that have a longer historical record: 1) ILI+, which estimates the proportion of outpatient doctor visits where the patient has influenza; and 2) rates of laboratory-confirmed influenza hospitalizations at a selected set of healthcare facilities. Our model, Flusion, is an ensemble that combines gradient boosting quantile regression models with a Bayesian autoregressive model. The gradient boosting models were trained on all three data signals, while the autoregressive model was trained on only the target signal; all models were trained jointly on data for multiple locations. Flusion was the top-performing model in the CDC's influenza prediction challenge for the 2023/24 season. In this article we investigate the factors contributing to Flusion's success, and we find that its strong performance was primarily driven by the use of a gradient boosting model that was trained jointly on data from multiple surveillance signals and locations. These results indicate the value of sharing information across locations and surveillance signals, especially when doing so adds to the pool of available training data.
△ Less
Submitted 26 July, 2024;
originally announced July 2024.
-
Evaluating infectious disease forecasts with allocation scoring rules
Authors:
Aaron Gerding,
Nicholas G. Reich,
Benjamin Rogers,
Evan L. Ray
Abstract:
Recent years have seen increasing efforts to forecast infectious disease burdens, with a primary goal being to help public health workers make informed policy decisions. However, there has only been limited discussion of how predominant forecast evaluation metrics might indicate the success of policies based in part on those forecasts. We explore one possible tether between forecasts and policy: t…
▽ More
Recent years have seen increasing efforts to forecast infectious disease burdens, with a primary goal being to help public health workers make informed policy decisions. However, there has only been limited discussion of how predominant forecast evaluation metrics might indicate the success of policies based in part on those forecasts. We explore one possible tether between forecasts and policy: the allocation of limited medical resources so as to minimize unmet need. We use probabilistic forecasts of disease burden in each of several regions to determine optimal resource allocations, and then we score forecasts according to how much unmet need their associated allocations would have allowed. We illustrate with forecasts of COVID-19 hospitalizations in the US, and we find that the forecast skill ranking given by this allocation scoring rule can vary substantially from the ranking given by the weighted interval score. We see this as evidence that the allocation scoring rule detects forecast value that is missed by traditional accuracy measures and that the general strategy of designing scoring rules that are directly linked to policy performance is a promising direction for epidemic forecast evaluation.
△ Less
Submitted 4 March, 2024; v1 submitted 23 December, 2023;
originally announced December 2023.
-
Comparison of Combination Methods to Create Calibrated Ensemble Forecasts for Seasonal Influenza in the U.S
Authors:
Nutcha Wattanachit,
Evan L. Ray,
Thomas C. McAndrew,
Nicholas G. Reich
Abstract:
The characteristics of influenza seasons varies substantially from year to year, posing challenges for public health preparation and response. Influenza forecasting is used to inform seasonal outbreak response, which can in turn potentially reduce the societal impact of an epidemic. The United States Centers for Disease Control and Prevention, in collaboration with external researchers, has run an…
▽ More
The characteristics of influenza seasons varies substantially from year to year, posing challenges for public health preparation and response. Influenza forecasting is used to inform seasonal outbreak response, which can in turn potentially reduce the societal impact of an epidemic. The United States Centers for Disease Control and Prevention, in collaboration with external researchers, has run an annual prospective influenza forecasting exercise, known as the FluSight challenge. A subset of participating teams has worked together to produce a collaborative multi-model ensemble, the FluSight Network ensemble. Uniting theoretical results from the forecasting literature with domain-specific forecasts from influenza outbreaks, we applied parametric forecast combination methods that simultaneously optimize individual model weights and calibrate the ensemble via a beta transformation. We used the beta-transformed linear pool and the finite beta mixture model to produce ensemble forecasts retrospectively for the 2016/2017 to 2018/2019 influenza seasons in the U.S. We compared their performance to methods currently used in the FluSight challenge, namely the equally weighted linear pool and the linear pool. Ensemble forecasts produced from methods with a beta transformation were shown to outperform those from the equally weighted linear pool and the linear pool for all week-ahead targets across in the test seasons based on average log scores. We observed improvements in overall accuracy despite the beta-transformed linear pool or beta mixture methods' modest under-prediction across all targets and seasons. Combination techniques that explicitly adjust for known calibration issues in linear pooling should be considered to improve ensemble probabilistic scores in outbreak settings.
△ Less
Submitted 15 March, 2022; v1 submitted 23 February, 2022;
originally announced February 2022.
-
Comparing trained and untrained probabilistic ensemble forecasts of COVID-19 cases and deaths in the United States
Authors:
Evan L. Ray,
Logan C. Brooks,
Jacob Bien,
Matthew Biggerstaff,
Nikos I. Bosse,
Johannes Bracher,
Estee Y. Cramer,
Sebastian Funk,
Aaron Gerding,
Michael A. Johansson,
Aaron Rumack,
Yijin Wang,
Martha Zorn,
Ryan J. Tibshirani,
Nicholas G. Reich
Abstract:
The U.S. COVID-19 Forecast Hub aggregates forecasts of the short-term burden of COVID-19 in the United States from many contributing teams. We study methods for building an ensemble that combines forecasts from these teams. These experiments have informed the ensemble methods used by the Hub. To be most useful to policy makers, ensemble forecasts must have stable performance in the presence of two…
▽ More
The U.S. COVID-19 Forecast Hub aggregates forecasts of the short-term burden of COVID-19 in the United States from many contributing teams. We study methods for building an ensemble that combines forecasts from these teams. These experiments have informed the ensemble methods used by the Hub. To be most useful to policy makers, ensemble forecasts must have stable performance in the presence of two key characteristics of the component forecasts: (1) occasional misalignment with the reported data, and (2) instability in the relative performance of component forecasters over time. Our results indicate that in the presence of these challenges, an untrained and robust approach to ensembling using an equally weighted median of all component forecasts is a good choice to support public health decision makers. In settings where some contributing forecasters have a stable record of good performance, trained ensembles that give those forecasters higher weight can also be helpful.
△ Less
Submitted 7 June, 2022; v1 submitted 28 January, 2022;
originally announced January 2022.
-
The Zoltar forecast archive: a tool to facilitate standardization and storage of interdisciplinary prediction research
Authors:
Nicholas G Reich,
Matthew Cornell,
Evan L Ray,
Katie House,
Khoa Le
Abstract:
Forecasting has emerged as an important component of informed, data-driven decision-making in a wide array of fields. We introduce a new data model for probabilistic predictions that encompasses a wide range of forecasting settings. This framework clearly defines the constituent parts of a probabilistic forecast and proposes one approach for representing these data elements. The data model is impl…
▽ More
Forecasting has emerged as an important component of informed, data-driven decision-making in a wide array of fields. We introduce a new data model for probabilistic predictions that encompasses a wide range of forecasting settings. This framework clearly defines the constituent parts of a probabilistic forecast and proposes one approach for representing these data elements. The data model is implemented in Zoltar, a new software application that stores forecasts using the data model and provides standardized API access to the data. In one real-time case study, an instance of the Zoltar web application was used to store, provide access to, and evaluate real-time forecast data on the order of 10$^7$ rows, provided by over 20 international research teams from academia and industry making forecasts of the COVID-19 outbreak in the US. Tools and data infrastructure for probabilistic forecasts, such as those introduced here, will play an increasingly important role in ensuring that future forecasting research adheres to a strict set of rigorous and reproducible standards.
△ Less
Submitted 6 June, 2020;
originally announced June 2020.
-
Evaluating epidemic forecasts in an interval format
Authors:
Johannes Bracher,
Evan L. Ray,
Tilmann Gneiting,
Nicholas G. Reich
Abstract:
For practical reasons, many forecasts of case, hospitalization and death counts in the context of the current COVID-19 pandemic are issued in the form of central predictive intervals at various levels. This is also the case for the forecasts collected in the COVID-19 Forecast Hub (https://covid19forecasthub.org/). Forecast evaluation metrics like the logarithmic score, which has been applied in se…
▽ More
For practical reasons, many forecasts of case, hospitalization and death counts in the context of the current COVID-19 pandemic are issued in the form of central predictive intervals at various levels. This is also the case for the forecasts collected in the COVID-19 Forecast Hub (https://covid19forecasthub.org/). Forecast evaluation metrics like the logarithmic score, which has been applied in several infectious disease forecasting challenges, are then not available as they require full predictive distributions. This article provides an overview of how established methods for the evaluation of quantile and interval forecasts can be applied to epidemic forecasts in this format. Specifically, we discuss the computation and interpretation of the weighted interval score, which is a proper score that approximates the continuous ranked probability score. It can be interpreted as a generalization of the absolute error to probabilistic forecasts and allows for a decomposition into a measure of sharpness and penalties for over- and underprediction.
△ Less
Submitted 8 January, 2021; v1 submitted 26 May, 2020;
originally announced May 2020.
-
Tuning Inelastic Light Scattering via Symmetry Control in 2D Magnet CrI$_3$
Authors:
Bevin Huang,
John Cenker,
Xiaoou Zhang,
Essance L. Ray,
Tiancheng Song,
Takashi Taniguchi,
Kenji Watanabe,
Michael A. McGuire,
Di Xiao,
Xiaodong Xu
Abstract:
The coupling between spin and charge degrees of freedom in a crystal imparts strong optical signatures on scattered electromagnetic waves. This has led to magneto-optical effects with a host of applications, from the sensitive detection of local magnetic order to optical modulation and data storage technologies. Here, we demonstrate a new magneto-optical effect, namely, the tuning of inelastically…
▽ More
The coupling between spin and charge degrees of freedom in a crystal imparts strong optical signatures on scattered electromagnetic waves. This has led to magneto-optical effects with a host of applications, from the sensitive detection of local magnetic order to optical modulation and data storage technologies. Here, we demonstrate a new magneto-optical effect, namely, the tuning of inelastically scattered light through symmetry control in atomically thin chromium triiodide (CrI$_3$). In monolayers, we found an extraordinarily large magneto-optical Raman effect from an A$_{1g}$ phonon mode due to the emergence of ferromagnetic order. The linearly polarized, inelastically scattered light rotates by ~40$^o$, more than two orders of magnitude larger than the rotation from MOKE under the same experimental conditions. In CrI$_3$ bilayers, we show that the same A$_{1g}$ phonon mode becomes Davydov-split into two modes of opposite parity, exhibiting divergent selection rules that depend on inversion symmetry and the underlying magnetic order. By switching between the antiferromagnetic states and the fully spin-polarized states with applied magnetic and electric fields, we demonstrate the magnetoelectrical control over their selection rules. Our work underscores the unique opportunities provided by 2D magnets for controlling the combined time-reversal and inversion symmetries to manipulate Raman optical selection rules and for exploring emergent magneto-optical effects and spin-phonon coupled physics.
△ Less
Submitted 21 November, 2019; v1 submitted 4 October, 2019;
originally announced October 2019.
-
Signatures of moiré-trapped valley excitons in MoSe$_2$/WSe$_2$ heterobilayers
Authors:
Kyle L. Seyler,
Pasqual Rivera,
Hongyi Yu,
Nathan P. Wilson,
Essance L. Ray,
David Mandrus,
Jiaqiang Yan,
Wang Yao,
Xiaodong Xu
Abstract:
The creation of moiré patterns in crystalline solids is a powerful approach to manipulate their electronic properties, which are fundamentally influenced by periodic potential landscapes. In 2D materials, a moiré pattern with a superlattice potential can form by vertically stacking two layered materials with a twist and/or finite lattice constant difference. This unique approach has led to emergen…
▽ More
The creation of moiré patterns in crystalline solids is a powerful approach to manipulate their electronic properties, which are fundamentally influenced by periodic potential landscapes. In 2D materials, a moiré pattern with a superlattice potential can form by vertically stacking two layered materials with a twist and/or finite lattice constant difference. This unique approach has led to emergent electronic phenomena, including the fractal quantum Hall effect, tunable Mott insulators, and unconventional superconductivity. Furthermore, theory predicts intriguing effects on optical excitations by a moiré potential in 2D valley semiconductors, but these signatures have yet to be experimentally detected. Here, we report experimental evidence of interlayer valley excitons trapped in a moiré potential in MoSe$_2$/WSe$_2$ heterobilayers. At low temperatures, we observe photoluminescence near the free interlayer exciton energy but with over 100 times narrower linewidths. The emitter g-factors are homogeneous across the same sample and only take two values, -15.9 and 6.7, in samples with twisting angles near 60° and 0°, respectively. The g-factors match those of the free interlayer exciton, which is determined by one of two possible valley pairing configurations. At a twist angle near 20°, the emitters become two orders of magnitude dimmer, but remarkably, they possess the same g-factor as the heterobilayer near 60°. This is consistent with the Umklapp recombination of interlayer excitons near the commensurate 21.8° twist angle. The emitters exhibit strong circular polarization, which implies the preservation of three-fold rotation symmetry by the trapping potential. Together with the power and excitation energy dependence, all evidence points to their origin as interlayer excitons trapped in a smooth moiré potential with inherited valley-contrasting physics.
△ Less
Submitted 12 September, 2018;
originally announced September 2018.
-
Prediction of infectious disease epidemics via weighted density ensembles
Authors:
Evan L. Ray,
Nicholas G. Reich
Abstract:
Accurate and reliable predictions of infectious disease dynamics can be valuable to public health organizations that plan interventions to decrease or prevent disease transmission. A great variety of models have been developed for this task, using different model structures, covariates, and targets for prediction. Experience has shown that the performance of these models varies; some tend to do be…
▽ More
Accurate and reliable predictions of infectious disease dynamics can be valuable to public health organizations that plan interventions to decrease or prevent disease transmission. A great variety of models have been developed for this task, using different model structures, covariates, and targets for prediction. Experience has shown that the performance of these models varies; some tend to do better or worse in different seasons or at different points within a season. Ensemble methods combine multiple models to obtain a single prediction that leverages the strengths of each model. We considered a range of ensemble methods that each form a predictive density for a target of interest as a weighted sum of the predictive densities from component models. In the simplest case, equal weight is assigned to each component model; in the most complex case, the weights vary with the region, prediction target, week of the season when the predictions are made, a measure of component model uncertainty, and recent observations of disease incidence. We applied these methods to predict measures of influenza season timing and severity in the United States, both at the national and regional levels, using three component models. We trained the models on retrospective predictions from 14 seasons (1997/1998 - 2010/2011) and evaluated each model's prospective, out-of-sample performance in the five subsequent influenza seasons. In this test phase, the ensemble methods showed overall performance that was similar to the best of the component models, but offered more consistent performance across seasons than the component models. Ensemble methods offer the potential to deliver more reliable predictions to public health decision makers.
△ Less
Submitted 31 March, 2017;
originally announced March 2017.