Progressive filling partitioning and mapping algorithm for Spark based on allocation fitness degree
The job execution mechanism of Spark was analyzed,task efficiency model and Shuffle model were established,then allocation fitness degree (AFD) was defined and the optimization goal was put forward.On the basis of the model definition,the progressive filling partitioning and mapping algorithm (PFPM)...
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Article |
| Language: | zho |
| Published: |
Editorial Department of Journal on Communications
2017-09-01
|
| Series: | Tongxin xuebao |
| Subjects: | |
| Online Access: | http://www.joconline.com.cn/zh/article/doi/10.11959/j.issn.1000-436x.2017188/ |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1841539490662842368 |
|---|---|
| author | Chen BIAN Jiong1 YU Wei-rong XIU Bin LIAO Chang-tian YING Yu-rong QIAN |
| author_facet | Chen BIAN Jiong1 YU Wei-rong XIU Bin LIAO Chang-tian YING Yu-rong QIAN |
| author_sort | Chen BIAN |
| collection | DOAJ |
| description | The job execution mechanism of Spark was analyzed,task efficiency model and Shuffle model were established,then allocation fitness degree (AFD) was defined and the optimization goal was put forward.On the basis of the model definition,the progressive filling partitioning and mapping algorithm (PFPM) was proposed.PFPM established the data distribution scheme adapting Reducers’ computing ability to decrease synchronous latency during Shuffle process and increase cluster the computing efficiency.The experiments demonstrate that PFPM could improve the rationality of workload distribution in Shuffle and optimize the execution efficiency of Spark. |
| format | Article |
| id | doaj-art-31df2c2fec8c40b187fb4f63eb22139e |
| institution | Kabale University |
| issn | 1000-436X |
| language | zho |
| publishDate | 2017-09-01 |
| publisher | Editorial Department of Journal on Communications |
| record_format | Article |
| series | Tongxin xuebao |
| spelling | doaj-art-31df2c2fec8c40b187fb4f63eb22139e2025-01-14T07:13:01ZzhoEditorial Department of Journal on CommunicationsTongxin xuebao1000-436X2017-09-013813314759712267Progressive filling partitioning and mapping algorithm for Spark based on allocation fitness degreeChen BIANJiong1 YUWei-rong XIUBin LIAOChang-tian YINGYu-rong QIANThe job execution mechanism of Spark was analyzed,task efficiency model and Shuffle model were established,then allocation fitness degree (AFD) was defined and the optimization goal was put forward.On the basis of the model definition,the progressive filling partitioning and mapping algorithm (PFPM) was proposed.PFPM established the data distribution scheme adapting Reducers’ computing ability to decrease synchronous latency during Shuffle process and increase cluster the computing efficiency.The experiments demonstrate that PFPM could improve the rationality of workload distribution in Shuffle and optimize the execution efficiency of Spark.http://www.joconline.com.cn/zh/article/doi/10.11959/j.issn.1000-436x.2017188/parallel computingSparkprogressive fillingpartitioning and mappingallocation fitness degree |
| spellingShingle | Chen BIAN Jiong1 YU Wei-rong XIU Bin LIAO Chang-tian YING Yu-rong QIAN Progressive filling partitioning and mapping algorithm for Spark based on allocation fitness degree Tongxin xuebao parallel computing Spark progressive filling partitioning and mapping allocation fitness degree |
| title | Progressive filling partitioning and mapping algorithm for Spark based on allocation fitness degree |
| title_full | Progressive filling partitioning and mapping algorithm for Spark based on allocation fitness degree |
| title_fullStr | Progressive filling partitioning and mapping algorithm for Spark based on allocation fitness degree |
| title_full_unstemmed | Progressive filling partitioning and mapping algorithm for Spark based on allocation fitness degree |
| title_short | Progressive filling partitioning and mapping algorithm for Spark based on allocation fitness degree |
| title_sort | progressive filling partitioning and mapping algorithm for spark based on allocation fitness degree |
| topic | parallel computing Spark progressive filling partitioning and mapping allocation fitness degree |
| url | http://www.joconline.com.cn/zh/article/doi/10.11959/j.issn.1000-436x.2017188/ |
| work_keys_str_mv | AT chenbian progressivefillingpartitioningandmappingalgorithmforsparkbasedonallocationfitnessdegree AT jiong1yu progressivefillingpartitioningandmappingalgorithmforsparkbasedonallocationfitnessdegree AT weirongxiu progressivefillingpartitioningandmappingalgorithmforsparkbasedonallocationfitnessdegree AT binliao progressivefillingpartitioningandmappingalgorithmforsparkbasedonallocationfitnessdegree AT changtianying progressivefillingpartitioningandmappingalgorithmforsparkbasedonallocationfitnessdegree AT yurongqian progressivefillingpartitioningandmappingalgorithmforsparkbasedonallocationfitnessdegree |