Main Image

PoS(CENet2017)035

A Load Balancing Strategy Based on Multiple Metrics in Spatial Image File Access Process in Hadoop

L. Yuan, Y. Feng, B. Li, F. Li

in The 7th International Conference on Computer Engineering and Networks

Contribution: pdf

Abstract

To solve the problem of load imbalance when accessing spatial image files in HDFS, we present a loading capacity model based on multiple attributes. In our model, five attributes, including
disk space load capacity, CPU processing capacity, memory processing capacity, disk read-write processing capacity and bandwidth, are introduced to measure the load balancing of each node. Then we used the objective weighting method of variation coefficient method to process experiment data, calculated the weight coefficient of the five attributes and obtained a load
capacity model. By this way, the nodes can calculate the actual load ability of their own, which will help improve the efficiency of the load balance when accessing spatial image files in HDFS.
To validate the proposed model, a large number of spatial image files are adopted to be accessed in our experiments. Experiment results show that, when accessing the spatial image files in HDFS, better load balance ability, system performance and efficiency can be obtained by using the strategy of the proposed multi-attribute load balance model.