Real-time data processing for serial crystallography experiments
We report the use of streaming data interfaces to perform fully online data processing for serial crystallography experiments, without storing intermediate data on disk. The system produces Bragg reflection intensity measurements suitable for scaling and merging, with a latency of less than 1 s per...
Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
International Union of Crystallography
2025-01-01
|
| Series: | IUCrJ |
| Subjects: | |
| Online Access: | https://journals.iucr.org/paper?S2052252524011837 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1841554732964904960 |
|---|---|
| author | Thomas White Tim Schoof Sergey Yakubov Aleksandra Tolstikova Philipp Middendorf Mikhail Karnevskiy Valerio Mariani Alessandra Henkel Bjarne Klopprogge Juergen Hannappel Dominik Oberthuer Ivan De Gennaro Aquino Dmitry Egorov Anna Munke Janina Sprenger Guillaume Pompidor Helena Taberman Andrey Gruzinov Jan Meyer Johanna Hakanpää Martin Gasthuber |
| author_facet | Thomas White Tim Schoof Sergey Yakubov Aleksandra Tolstikova Philipp Middendorf Mikhail Karnevskiy Valerio Mariani Alessandra Henkel Bjarne Klopprogge Juergen Hannappel Dominik Oberthuer Ivan De Gennaro Aquino Dmitry Egorov Anna Munke Janina Sprenger Guillaume Pompidor Helena Taberman Andrey Gruzinov Jan Meyer Johanna Hakanpää Martin Gasthuber |
| author_sort | Thomas White |
| collection | DOAJ |
| description | We report the use of streaming data interfaces to perform fully online data processing for serial crystallography experiments, without storing intermediate data on disk. The system produces Bragg reflection intensity measurements suitable for scaling and merging, with a latency of less than 1 s per frame. Our system uses the CrystFEL software in combination with the ASAP::O data framework. In a series of user experiments at PETRA III, frames from a 16 megapixel Dectris EIGER2 X detector were searched for peaks, indexed and integrated at the maximum full-frame readout speed of 133 frames per second. The computational resources required depend on various factors, most significantly the fraction of non-blank frames (`hits'). The average single-thread processing time per frame was 242 ms for blank frames and 455 ms for hits, meaning that a single 96-core computing node was sufficient to keep up with the data, with ample headroom for unexpected throughput reductions. Further significant improvements are expected, for example by binning pixel intensities together to reduce the pixel count. We discuss the implications of real-time data processing on the `data deluge' problem from recent and future photon-science experiments, in particular on calibration requirements, computing access patterns and the need for the preservation of raw data. |
| format | Article |
| id | doaj-art-cd282ae9fea2420783feea606c764bfc |
| institution | Kabale University |
| issn | 2052-2525 |
| language | English |
| publishDate | 2025-01-01 |
| publisher | International Union of Crystallography |
| record_format | Article |
| series | IUCrJ |
| spelling | doaj-art-cd282ae9fea2420783feea606c764bfc2025-01-08T10:32:13ZengInternational Union of CrystallographyIUCrJ2052-25252025-01-011219710810.1107/S2052252524011837if5005Real-time data processing for serial crystallography experimentsThomas White0Tim Schoof1Sergey Yakubov2Aleksandra Tolstikova3Philipp Middendorf4Mikhail Karnevskiy5Valerio Mariani6Alessandra Henkel7Bjarne Klopprogge8Juergen Hannappel9Dominik Oberthuer10Ivan De Gennaro Aquino11Dmitry Egorov12Anna Munke13Janina Sprenger14Guillaume Pompidor15Helena Taberman16Andrey Gruzinov17Jan Meyer18Johanna Hakanpää19Martin Gasthuber20Center for Data and Computing in Natural Science CDCS, Deutsches Elektronen-Synchrotron DESY, Hamburg, GermanyDeutsches Elektronen-Synchrotron DESY, Hamburg, GermanyDeutsches Elektronen-Synchrotron DESY, Hamburg, GermanyDeutsches Elektronen-Synchrotron DESY, Hamburg, GermanyCenter for Free-Electron Laser Science CFEL, Deutsches Elektronen-Synchrotron DESY, Hamburg, GermanyDeutsches Elektronen-Synchrotron DESY, Hamburg, GermanyLinac Coherent Light Source, SLAC National Accelerator Laboratory, Menlo Park, USACenter for Free-Electron Laser Science CFEL, Deutsches Elektronen-Synchrotron DESY, Hamburg, GermanyCenter for Free-Electron Laser Science CFEL, Deutsches Elektronen-Synchrotron DESY, Hamburg, GermanyDeutsches Elektronen-Synchrotron DESY, Hamburg, GermanyCenter for Free-Electron Laser Science CFEL, Deutsches Elektronen-Synchrotron DESY, Hamburg, GermanyCenter for Free-Electron Laser Science CFEL, Deutsches Elektronen-Synchrotron DESY, Hamburg, GermanyCenter for Free-Electron Laser Science CFEL, Deutsches Elektronen-Synchrotron DESY, Hamburg, GermanyCenter for Free-Electron Laser Science CFEL, Deutsches Elektronen-Synchrotron DESY, Hamburg, GermanyCenter for Free-Electron Laser Science CFEL, Deutsches Elektronen-Synchrotron DESY, Hamburg, GermanyDeutsches Elektronen-Synchrotron DESY, Hamburg, GermanyDeutsches Elektronen-Synchrotron DESY, Hamburg, GermanyDeutsches Elektronen-Synchrotron DESY, Hamburg, GermanyDeutsches Elektronen-Synchrotron DESY, Hamburg, GermanyDeutsches Elektronen-Synchrotron DESY, Hamburg, GermanyDeutsches Elektronen-Synchrotron DESY, Hamburg, GermanyWe report the use of streaming data interfaces to perform fully online data processing for serial crystallography experiments, without storing intermediate data on disk. The system produces Bragg reflection intensity measurements suitable for scaling and merging, with a latency of less than 1 s per frame. Our system uses the CrystFEL software in combination with the ASAP::O data framework. In a series of user experiments at PETRA III, frames from a 16 megapixel Dectris EIGER2 X detector were searched for peaks, indexed and integrated at the maximum full-frame readout speed of 133 frames per second. The computational resources required depend on various factors, most significantly the fraction of non-blank frames (`hits'). The average single-thread processing time per frame was 242 ms for blank frames and 455 ms for hits, meaning that a single 96-core computing node was sufficient to keep up with the data, with ample headroom for unexpected throughput reductions. Further significant improvements are expected, for example by binning pixel intensities together to reduce the pixel count. We discuss the implications of real-time data processing on the `data deluge' problem from recent and future photon-science experiments, in particular on calibration requirements, computing access patterns and the need for the preservation of raw data.https://journals.iucr.org/paper?S2052252524011837macromolecular crystallographyserial crystallographyx-ray crystallographyreal-time data processing |
| spellingShingle | Thomas White Tim Schoof Sergey Yakubov Aleksandra Tolstikova Philipp Middendorf Mikhail Karnevskiy Valerio Mariani Alessandra Henkel Bjarne Klopprogge Juergen Hannappel Dominik Oberthuer Ivan De Gennaro Aquino Dmitry Egorov Anna Munke Janina Sprenger Guillaume Pompidor Helena Taberman Andrey Gruzinov Jan Meyer Johanna Hakanpää Martin Gasthuber Real-time data processing for serial crystallography experiments IUCrJ macromolecular crystallography serial crystallography x-ray crystallography real-time data processing |
| title | Real-time data processing for serial crystallography experiments |
| title_full | Real-time data processing for serial crystallography experiments |
| title_fullStr | Real-time data processing for serial crystallography experiments |
| title_full_unstemmed | Real-time data processing for serial crystallography experiments |
| title_short | Real-time data processing for serial crystallography experiments |
| title_sort | real time data processing for serial crystallography experiments |
| topic | macromolecular crystallography serial crystallography x-ray crystallography real-time data processing |
| url | https://journals.iucr.org/paper?S2052252524011837 |
| work_keys_str_mv | AT thomaswhite realtimedataprocessingforserialcrystallographyexperiments AT timschoof realtimedataprocessingforserialcrystallographyexperiments AT sergeyyakubov realtimedataprocessingforserialcrystallographyexperiments AT aleksandratolstikova realtimedataprocessingforserialcrystallographyexperiments AT philippmiddendorf realtimedataprocessingforserialcrystallographyexperiments AT mikhailkarnevskiy realtimedataprocessingforserialcrystallographyexperiments AT valeriomariani realtimedataprocessingforserialcrystallographyexperiments AT alessandrahenkel realtimedataprocessingforserialcrystallographyexperiments AT bjarneklopprogge realtimedataprocessingforserialcrystallographyexperiments AT juergenhannappel realtimedataprocessingforserialcrystallographyexperiments AT dominikoberthuer realtimedataprocessingforserialcrystallographyexperiments AT ivandegennaroaquino realtimedataprocessingforserialcrystallographyexperiments AT dmitryegorov realtimedataprocessingforserialcrystallographyexperiments AT annamunke realtimedataprocessingforserialcrystallographyexperiments AT janinasprenger realtimedataprocessingforserialcrystallographyexperiments AT guillaumepompidor realtimedataprocessingforserialcrystallographyexperiments AT helenataberman realtimedataprocessingforserialcrystallographyexperiments AT andreygruzinov realtimedataprocessingforserialcrystallographyexperiments AT janmeyer realtimedataprocessingforserialcrystallographyexperiments AT johannahakanpaa realtimedataprocessingforserialcrystallographyexperiments AT martingasthuber realtimedataprocessingforserialcrystallographyexperiments |