Data driven weather forecasts trained and initialised directly from observations
Authors:
Anthony McNally,
Christian Lessig,
Peter Lean,
Eulalie Boucher,
Mihai Alexe,
Ewan Pinnington,
Matthew Chantry,
Simon Lang,
Chris Burrows,
Marcin Chrust,
Florian Pinault,
Ethel Villeneuve,
Niels Bormann,
Sean Healy
Abstract:
Skilful Machine Learned weather forecasts have challenged our approach to numerical weather prediction, demonstrating competitive performance compared to traditional physics-based approaches. Data-driven systems have been trained to forecast future weather by learning from long historical records of past weather such as the ECMWF ERA5. These datasets have been made freely available to the wider re…
▽ More
Skilful Machine Learned weather forecasts have challenged our approach to numerical weather prediction, demonstrating competitive performance compared to traditional physics-based approaches. Data-driven systems have been trained to forecast future weather by learning from long historical records of past weather such as the ECMWF ERA5. These datasets have been made freely available to the wider research community, including the commercial sector, which has been a major factor in the rapid rise of ML forecast systems and the levels of accuracy they have achieved. However, historical reanalyses used for training and real-time analyses used for initial conditions are produced by data assimilation, an optimal blending of observations with a physics-based forecast model. As such, many ML forecast systems have an implicit and unquantified dependence on the physics-based models they seek to challenge. Here we propose a new approach, training a neural network to predict future weather purely from historical observations with no dependence on reanalyses. We use raw observations to initialise a model of the atmosphere (in observation space) learned directly from the observations themselves. Forecasts of crucial weather parameters (such as surface temperature and wind) are obtained by predicting weather parameter observations (e.g. SYNOP surface data) at future times and arbitrary locations. We present preliminary results on forecasting observations 12-hours into the future. These already demonstrate successful learning of time evolutions of the physical processes captured in real observations. We argue that this new approach, by staying purely in observation space, avoids many of the challenges of traditional data assimilation, can exploit a wider range of observations and is readily expanded to simultaneous forecasting of the full Earth system (atmosphere, land, ocean and composition).
△ Less
Submitted 22 July, 2024;
originally announced July 2024.
A Scalable MCEM Estimator for Spatio-Temporal Autoregressive Models
Authors:
Philipp Hunziker,
Julian Wucherpfennig,
Aya Kachi,
Nils-Christian Bormann
Abstract:
Very large spatio-temporal lattice data are becoming increasingly common across a variety of disciplines. However, estimating interdependence across space and time in large areal datasets remains challenging, as existing approaches are often (i) not scalable, (ii) designed for conditionally Gaussian outcome data, or (iii) are limited to cross-sectional and univariate outcomes. This paper proposes…
▽ More
Very large spatio-temporal lattice data are becoming increasingly common across a variety of disciplines. However, estimating interdependence across space and time in large areal datasets remains challenging, as existing approaches are often (i) not scalable, (ii) designed for conditionally Gaussian outcome data, or (iii) are limited to cross-sectional and univariate outcomes. This paper proposes an MCEM estimation strategy for a family of latent-Gaussian multivariate spatio-temporal models that addresses these issues. The proposed estimator is applicable to a wide range of non-Gaussian outcomes, and implementations for binary and count outcomes are discussed explicitly. The methodology is illustrated on simulated data, as well as on weekly data of IS-related events in Syrian districts.
△ Less
Submitted 18 July, 2018;
originally announced July 2018.
