A Big Data Platform for heterogeneous data collection and analysis in large-scale data centres
October 22, 2021
The INFN-CNAF data centre hosts the Italian Tier~1 site for the Worldwide LHC Computing Grid (WLCG), while also serving several other research and technological transfer programs. The challenges posed by the upcoming runs of LHC, together with the opportunity of moving the data centre itself to a bigger site, require a thorough redesign of its monitoring system. The large but heterogeneous amount of logging data and metrics produced daily are fundamental for monitoring activities and, once harmonised, can also be used to build Predictive Maintenance models based on Big Data techniques.
In this work we describe the Big Data Platform, a new monitoring infrastructure under development at CNAF. The Big Data Platform relies on a modular, highly scalable architecture based on open source technologies and able to exploit modern frameworks such as containerisation and cloud support.
It is capable of collecting data from heterogeneous data sources, clean and harmonise them, and store them as JSON files on different solutions, based on the needs of the end user. Data can then be visualised using Kibana, or analysed through a platform based on Jupyter Notebooks.
How to cite
Metadata are provided both in "article" format (very similar to INSPIRE) as this helps creating
very compact bibliographies which can be beneficial to authors and
readers, and in "proceeding" format
which is more detailed and complete.