PoS - Proceedings of Science
Volume 414 - 41st International Conference on High Energy physics (ICHEP2022) - Computing and Data Handling
An intelligent Data Delivery Service for and beyond the ATLAS experiment
W. Guan*, T. Maeno, B. Paul Bockelman, T. Wenaus, R. Zhang, C. Weber, F. Harald Barreiro Megino, F. Lin and A. Alekseev
Full text: pdf
Pre-published on: November 21, 2022
Published on: June 15, 2023
Abstract
The intelligent Data Delivery Service (iDDS) has been developed to cope with the huge increase of computing and storage resource usage in the coming LHC data taking. It has been designed to intelligently orchestrate workflows and data management systems, decoupling data pre-processing, delivery, and primary processing in large scale workflows. It is an experiment-agnostic service that has been deployed to serve data carousel (orchestrating efficient processing of tape-resident data), machine learning hyperparameter optimization, active learning, and other complex multi-stage workflows defined via DAG (Directed Acyclic Graph), CWL (Common Workflow Language) and other descriptions, including a growing number of analysis workflows. We will at first introduce some deployed use cases in a summary. Then we will focus on new improvements and use cases under developments in ATLAS, Rubin Observatory and sPHENIX, together with future efforts.
DOI: https://doi.org/10.22323/1.414.0218
How to cite

Metadata are provided both in "article" format (very similar to INSPIRE) as this helps creating very compact bibliographies which can be beneficial to authors and readers, and in "proceeding" format which is more detailed and complete.

Open Access
Creative Commons LicenseCopyright owned by the author(s) under the term of the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.