Main Image
Volume 282 - 38th International Conference on High Energy Physics (ICHEP2016) - Computing
Developments in Architectures and Services for using High Performance Computing in Energy Frontier Experiments
J.T. Childers,* L. Gerhardt, D. Bard, D. Benjamin, W. Bhimji, S. Farrell, M. Fasel, T. LeCompte, J. Porter, V. Tsulaia, T. Uram Prabhat
*corresponding author
Full text: pdf
Pre-published on: 2017-02-06 14:28:00
Published on: 2017-04-19 16:48:57
Abstract
The integration of HPC resources into the standard computing toolkit of HEP experiments is becoming important as the needs of experiments outpace traditional resources. We will describe solutions that address some of the difficulty in running data-intensive pipelines on HPC systems. Users of NERSC HPCs are benefiting from a newly developed package called "Shifter" that creates Docker-like images and the deployment of the new "Burst Buffers" NVRAM file system designed to offer extreme I/O performance, supporting terabyte-per-second bandwidth and over 10 million IO operations per second. These tools have enabled particle physicists from multiple experiments to routinely run their entire multi-TB CVMFS-based software stacks across tens of thousands of compute cores. In addition, an Edge Service has been developed to provide a uniform interface for HEP job management systems to access supercomputers. It is based on the Python Django framework and is composed of two processes, of which one runs inside the supercomputing environment and one runs outside. It has been used to run over 100 million core-hours of LHC experiment jobs on the Mira supercomputer at the Argonne Leadership Computing Facility.
Open Access
Creative Commons LicenseCopyright owned by the author(s) under the term of the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.