PoS - Proceedings of Science
Volume 378 - International Symposium on Grids & Clouds 2021 (ISGC2021) - Data Management & Big Data
A Big Data Platform for heterogeneous data collection and analysis in large-scale data centres
S. Rossi Tisbeni*, D. Cesini, B. Martelli, A. Carbone, C. Cavallaro, D.C. Duma, A. Falabella, M. Galletti, J. Gasparetto, E. Furlan, D. Michelotto, F. Minarini, L. Morganti, E. Ronchieri and G. Sergi
Full text: pdf
Published on: October 22, 2021
Abstract
The INFN-CNAF data centre hosts the Italian Tier~1 site for the Worldwide LHC Computing Grid (WLCG), while also serving several other research and technological transfer programs. The challenges posed by the upcoming runs of LHC, together with the opportunity of moving the data centre itself to a bigger site, require a thorough redesign of its monitoring system. The large but heterogeneous amount of logging data and metrics produced daily are fundamental for monitoring activities and, once harmonised, can also be used to build Predictive Maintenance models based on Big Data techniques.

In this work we describe the Big Data Platform, a new monitoring infrastructure under development at CNAF. The Big Data Platform relies on a modular, highly scalable architecture based on open source technologies and able to exploit modern frameworks such as containerisation and cloud support.
It is capable of collecting data from heterogeneous data sources, clean and harmonise them, and store them as JSON files on different solutions, based on the needs of the end user. Data can then be visualised using Kibana, or analysed through a platform based on Jupyter Notebooks.
DOI: https://doi.org/10.22323/1.378.0008
How to cite

Metadata are provided both in "article" format (very similar to INSPIRE) as this helps creating very compact bibliographies which can be beneficial to authors and readers, and in "proceeding" format which is more detailed and complete.

Open Access
Creative Commons LicenseCopyright owned by the author(s) under the term of the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.