Distributed storage systems usually adopt replication for reliability and fast access. However, as the data volume grows, many large-scale storage systems are tending to employ erasurv coding to reduce the storage ove...Distributed storage systems usually adopt replication for reliability and fast access. However, as the data volume grows, many large-scale storage systems are tending to employ erasurv coding to reduce the storage overhead of replication while deliver the same reliabil- ity. Unfortunately, erasure coding could result in performance degrada- tion due to less data locality and degraded reads. To trade off among reli- ability, performance and storage overhead at the same time, we propose FLEXBal, a flexible dual-scheme block management approach. FLEXBal supports both replication and erasure coding simultaneously, and applies them dynamically according to the recent data temperature. Erase cod- ing is for cold data to reduce storage, while replication is for hot files so that applications can leverage data locality. To guarantee the same reli- ability as replication with fewer replicas, FLEXBal models block place- ment with bipartite graphs. The prototype of FLEXBal is implemented based on HDFS. The experimental results show that FLEXBal succeeds in reducing the storage overhead even for a scenario with many smal1 files without reliability compromising, and meanwhile, providing better data locality for frequently accessed datasets.展开更多
摘要Distributed storage systems usually adopt replication for reliability and fast access. However, as the data volume grows, many large-scale storage systems are tending to employ erasurv coding to reduce the storage overhead of replication while deliver the same reliabil- ity. Unfortunately, erasure coding could result in performance degrada- tion due to less data locality and degraded reads. To trade off among reli- ability, performance and storage overhead at the same time, we propose FLEXBal, a flexible dual-scheme block management approach. FLEXBal supports both replication and erasure coding simultaneously, and applies them dynamically according to the recent data temperature. Erase cod- ing is for cold data to reduce storage, while replication is for hot files so that applications can leverage data locality. To guarantee the same reli- ability as replication with fewer replicas, FLEXBal models block place- ment with bipartite graphs. The prototype of FLEXBal is implemented based on HDFS. The experimental results show that FLEXBal succeeds in reducing the storage overhead even for a scenario with many smal1 files without reliability compromising, and meanwhile, providing better data locality for frequently accessed datasets.