Scalable data assimilation with message passing
Data assimilation is a core component of numerical weather prediction systems. The large quantity of data processed during assimilation requires the computation to be distributed across increasingly many compute nodes; yet, existing approaches suffer from synchronization overhead in this setting. In...
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Cambridge University Press
2025-01-01
|
| Series: | Environmental Data Science |
| Subjects: | |
| Online Access: | https://www.cambridge.org/core/product/identifier/S2634460224000475/type/journal_article |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1841526509142016000 |
|---|---|
| author | Oscar Key So Takao Daniel Giles Marc Peter Deisenroth |
| author_facet | Oscar Key So Takao Daniel Giles Marc Peter Deisenroth |
| author_sort | Oscar Key |
| collection | DOAJ |
| description | Data assimilation is a core component of numerical weather prediction systems. The large quantity of data processed during assimilation requires the computation to be distributed across increasingly many compute nodes; yet, existing approaches suffer from synchronization overhead in this setting. In this article, we exploit the formulation of data assimilation as a Bayesian inference problem and apply a message-passing algorithm to solve the spatial inference problem. Since message passing is inherently based on local computations, this approach lends itself to parallel and distributed computation. In combination with a GPU-accelerated implementation, we can scale the algorithm to very large grid sizes while retaining good accuracy and compute and memory requirements. |
| format | Article |
| id | doaj-art-937e827adda444ca8fe2bf0fa7d670b8 |
| institution | Kabale University |
| issn | 2634-4602 |
| language | English |
| publishDate | 2025-01-01 |
| publisher | Cambridge University Press |
| record_format | Article |
| series | Environmental Data Science |
| spelling | doaj-art-937e827adda444ca8fe2bf0fa7d670b82025-01-16T21:47:30ZengCambridge University PressEnvironmental Data Science2634-46022025-01-01410.1017/eds.2024.47Scalable data assimilation with message passingOscar Key0https://orcid.org/0009-0009-1357-471XSo Takao1Daniel Giles2https://orcid.org/0000-0002-3668-1851Marc Peter Deisenroth3https://orcid.org/0000-0003-1503-680XUCL Centre for Artificial Intelligence, University College London, London, United KingdomUCL Centre for Artificial Intelligence, University College London, London, United Kingdom Department of Computing and Mathematical Sciences, California Institute of Technology, Pasadena, CA, United StatesUCL Centre for Artificial Intelligence, University College London, London, United KingdomUCL Centre for Artificial Intelligence, University College London, London, United Kingdom The Alan Turing Institute, London, United KingdomData assimilation is a core component of numerical weather prediction systems. The large quantity of data processed during assimilation requires the computation to be distributed across increasingly many compute nodes; yet, existing approaches suffer from synchronization overhead in this setting. In this article, we exploit the formulation of data assimilation as a Bayesian inference problem and apply a message-passing algorithm to solve the spatial inference problem. Since message passing is inherently based on local computations, this approach lends itself to parallel and distributed computation. In combination with a GPU-accelerated implementation, we can scale the algorithm to very large grid sizes while retaining good accuracy and compute and memory requirements.https://www.cambridge.org/core/product/identifier/S2634460224000475/type/journal_articleBayesian inferencedata assimilationdistributed computationmessage passing |
| spellingShingle | Oscar Key So Takao Daniel Giles Marc Peter Deisenroth Scalable data assimilation with message passing Environmental Data Science Bayesian inference data assimilation distributed computation message passing |
| title | Scalable data assimilation with message passing |
| title_full | Scalable data assimilation with message passing |
| title_fullStr | Scalable data assimilation with message passing |
| title_full_unstemmed | Scalable data assimilation with message passing |
| title_short | Scalable data assimilation with message passing |
| title_sort | scalable data assimilation with message passing |
| topic | Bayesian inference data assimilation distributed computation message passing |
| url | https://www.cambridge.org/core/product/identifier/S2634460224000475/type/journal_article |
| work_keys_str_mv | AT oscarkey scalabledataassimilationwithmessagepassing AT sotakao scalabledataassimilationwithmessagepassing AT danielgiles scalabledataassimilationwithmessagepassing AT marcpeterdeisenroth scalabledataassimilationwithmessagepassing |