Volume 476 - 42nd International Conference on High Energy Physics (ICHEP2024) - Computing and Data Handling Posters
Design of a Distributed Parallel Computing Engine for High-Throughput Data Processing of HEPS
Y. Hu*, R. Liu, S. Fu and F. Qi
*: corresponding author
Full text: pdf
Pre-published on: January 07, 2025
Published on: April 29, 2025
Abstract
HEPS is a fourth-generation synchrotron light source, and the experiments conducted at HEPS will transition to high-throughput, multi-modal, ultra-fast frequency, and cross-scale formats. The annual data flux generated by these experiments is anticipated to enter the ’Exa-scale’ era. Given the substantial volume of high-throughput experimental data, a single computing node struggles to meet the computational demands for data analysis. Consequently, it is essential to develop a high-performance, robust, and user-friendly distributed parallel computing engine to enhance the performance of data processing software. Due to significant variations in data rates across different beamline stations, supporting heterogeneous resources (such as GPUs) in a flexible and fine-grained manner presents a challenge. To optimize computational efficiency and handle extensive datasets, we have developed a distributed parallel computing engine that can leverage
scalable, heterogeneous computing resources to deliver HEPS’ data analytical services across
various scales. Experiment shows that our distributed computing engine significantly enhances
the efficiency of HEPS’ data processing.
DOI: https://doi.org/10.22323/1.476.1035
How to cite

Metadata are provided both in article format (very similar to INSPIRE) as this helps creating very compact bibliographies which can be beneficial to authors and readers, and in proceeding format which is more detailed and complete.

Open Access
Creative Commons LicenseCopyright owned by the author(s) under the term of the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.