Efficient Process Mapping in Geo-Distributed Cloud Data Centers

International Conference for High Performance Computing, Networking, Storage, and Analysis (SC)

Amelie Chi Zhou¹    Yifan Gong²    Bingsheng He³    Jidong Zhai⁴

¹Shenzhen University    ²TuSimple    ³National University of Singapore     ⁴Tsinghua University

Abstract

Recently, various applications including data analytics and machine learning have been developed for geo-distributed cloud data centers. For those applications, the ways to map parallel processes to physical nodes (i.e., “process mapping”) could significantly impact the performance of the applications because of non-uniform communication cost in such geo-distributed environments. While process mapping has been widely studied in grid/cluster environments, few of the existing studies have considered the problem in geo-distributed cloud environments. In this paper, we propose a novel model to formulate the geo-distributed process mapping problem and develop a new method to efficiently find the near optimal solution. Our algorithm considers both the network communication performance of geo-distributed data centers as well as the communication matrix of the target application. Evaluation results with real experiments on Amazon EC2 and simulations demonstrate that our proposal achieves significant performance improvement (50% on average) compared to the state-of-the-art algorithms.

Figure 1: Geographic distributions of Amazon EC2 regions.

Figure 3: Communication pattern matrices. The color scale indicates communication volume in Bytes (darker means heavier traffic).

Figure 4: Overhead of compared algorithms under different scales (normalized to Baseline)

Figure 5: Overall performance improvement for the five appli-cations on Amazon EC2 (normalized to the average of Baseline)

Figure 6: Overall communication performance improvement comparison for the five applications on simulation (normalized to the average of Baseline)

Figure 7: Performance improvement in different scales (in numbers of machines)

Acknowledgements

We would like to thank Prof. Amy Apon for shepherding the paper and anonymous reviewers from SC17. This work is supported by a MoE AcRF Tier 1 grant (T1 251RES1610) in Singapore, a collaborative grant from Microsoft Research Asia and NSFC Project 61628204 in China. Jidong Zhai’s research is partly supported by National Key Research and Development Program of China 2016YFB0200100 and NSFC Project 61472201. Yifan Gong’s research is partly supported by the HPC Group of TuSimple. Amelie Chi Zhou’s work is partly supported by grant NSFC-Guangdong U1301252. Amelie Chi Zhou, Bingsheng He and Jidong Zhai are corresponding authors of the paper.

Bibtex

@inproceedings{ProcessMapping,

author = {Zhou, Amelie Chi and Gong, Yifan and He, Bingsheng and Zhai, Jidong},

title = {Efficient Process Mapping in Geo-Distributed Cloud Data Centers},

year = {2017},

articleno = {16},

numpages = {12},

series = {SC '17}

}

Downloads