TY - GEN
T1 - Correlation based file prefetching approach for Hadoop
AU - Dong, Bo
AU - Zhong, Xiao
AU - Zheng, Qinghua
AU - Jian, Lirong
AU - Liu, Jian
AU - Qiu, Jie
AU - Li, Ying
PY - 2010
Y1 - 2010
N2 - Hadoop Distributed File System (HDFS) has been widely adopted to support Internet applications because of its reliable, scalable and low-cost storage capability. BlueSky, one of the most popular e-Learning resource sharing systems in China, is utilizing HDFS to store massive courseware. However, due to the inefficient access mechanism of HDFS, access latency of reading files from HDFS significantly impacts the performance of processing user requests. This paper introduces a two-level correlation based file prefetching approach, taking the characteristics of HDFS into consideration, to improve performance by reducing access latency. Four placement patterns to store prefetched data are presented, with policies to achieve trade-off between performance and efficiency of HDFS prefetching. Moreover, a dynamic replica selection algorithm is investigated to improve the efficiency of HDFS prefetching. The proposed prefetching approach has been implemented in BlueSky, and experimental results prove that correlation based file prefetching can significantly reduce access latency therefore improve performance of Hadoop-based Internet applications.
AB - Hadoop Distributed File System (HDFS) has been widely adopted to support Internet applications because of its reliable, scalable and low-cost storage capability. BlueSky, one of the most popular e-Learning resource sharing systems in China, is utilizing HDFS to store massive courseware. However, due to the inefficient access mechanism of HDFS, access latency of reading files from HDFS significantly impacts the performance of processing user requests. This paper introduces a two-level correlation based file prefetching approach, taking the characteristics of HDFS into consideration, to improve performance by reducing access latency. Four placement patterns to store prefetched data are presented, with policies to achieve trade-off between performance and efficiency of HDFS prefetching. Moreover, a dynamic replica selection algorithm is investigated to improve the efficiency of HDFS prefetching. The proposed prefetching approach has been implemented in BlueSky, and experimental results prove that correlation based file prefetching can significantly reduce access latency therefore improve performance of Hadoop-based Internet applications.
KW - Cloud storage
KW - File correlation
KW - Hadoop distributed file system
KW - Prefetching
UR - https://www.scopus.com/pages/publications/79952379409
U2 - 10.1109/CloudCom.2010.60
DO - 10.1109/CloudCom.2010.60
M3 - 会议稿件
AN - SCOPUS:79952379409
SN - 9780769543024
T3 - Proceedings - 2nd IEEE International Conference on Cloud Computing Technology and Science, CloudCom 2010
SP - 41
EP - 48
BT - Proceedings - 2nd IEEE International Conference on Cloud Computing Technology and Science, CloudCom 2010
PB - IEEE Computer Society
T2 - 2nd IEEE International Conference on Cloud Computing Technology and Science, CloudCom 2010
Y2 - 30 November 2010 through 3 December 2010
ER -