Load prediction based elastic resource scheduling strategy in Flink
In order to solve the problem that the load of big data stream computing platform fluctuates drastically while the cluster was suffering from the performance bottleneck due to the shortage of computing resources,the load prediction based elastic resource scheduling strategy in Flink (LPERS-Flink) wa...
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Article |
| Language: | zho |
| Published: |
Editorial Department of Journal on Communications
2020-10-01
|
| Series: | Tongxin xuebao |
| Subjects: | |
| Online Access: | http://www.joconline.com.cn/zh/article/doi/10.11959/j.issn.1000-436x.2020195/ |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1841539263398674432 |
|---|---|
| author | Ziyang LI Jiong YU Yuefei WANG Chen BIAN Yonglin PU Yitian ZHANG Yu LIU |
| author_facet | Ziyang LI Jiong YU Yuefei WANG Chen BIAN Yonglin PU Yitian ZHANG Yu LIU |
| author_sort | Ziyang LI |
| collection | DOAJ |
| description | In order to solve the problem that the load of big data stream computing platform fluctuates drastically while the cluster was suffering from the performance bottleneck due to the shortage of computing resources,the load prediction based elastic resource scheduling strategy in Flink (LPERS-Flink) was proposed.Firstly,the load prediction model was set up as the foundation to propose the load prediction algorithm and predict the variation tendency of the processing load.Secondly,the resource judgment model was set up to identify the performance bottleneck and resource redundancy of the cluster while the resource scheduling algorithm was proposed to draw up the resource rescheduling plan.Finally,the online load migration algorithm was proposed to execute the resource rescheduling plan and migrate processing load among nodes efficiently.The experimental results show that the strategy provides better performance promotion in the application with drastically fluctuating processing load.The scale and resource configuration of the cluster responded to the variation of processing load in time and the communication overhead of the load migration was reduced effectively. |
| format | Article |
| id | doaj-art-2309dc0e7c874feb8ca9fa735e4ccd48 |
| institution | Kabale University |
| issn | 1000-436X |
| language | zho |
| publishDate | 2020-10-01 |
| publisher | Editorial Department of Journal on Communications |
| record_format | Article |
| series | Tongxin xuebao |
| spelling | doaj-art-2309dc0e7c874feb8ca9fa735e4ccd482025-01-14T07:20:06ZzhoEditorial Department of Journal on CommunicationsTongxin xuebao1000-436X2020-10-01419210859737846Load prediction based elastic resource scheduling strategy in FlinkZiyang LIJiong YUYuefei WANGChen BIANYonglin PUYitian ZHANGYu LIUIn order to solve the problem that the load of big data stream computing platform fluctuates drastically while the cluster was suffering from the performance bottleneck due to the shortage of computing resources,the load prediction based elastic resource scheduling strategy in Flink (LPERS-Flink) was proposed.Firstly,the load prediction model was set up as the foundation to propose the load prediction algorithm and predict the variation tendency of the processing load.Secondly,the resource judgment model was set up to identify the performance bottleneck and resource redundancy of the cluster while the resource scheduling algorithm was proposed to draw up the resource rescheduling plan.Finally,the online load migration algorithm was proposed to execute the resource rescheduling plan and migrate processing load among nodes efficiently.The experimental results show that the strategy provides better performance promotion in the application with drastically fluctuating processing load.The scale and resource configuration of the cluster responded to the variation of processing load in time and the communication overhead of the load migration was reduced effectively.http://www.joconline.com.cn/zh/article/doi/10.11959/j.issn.1000-436x.2020195/stream computingresource schedulingload predictionperformance bottleneckFlink |
| spellingShingle | Ziyang LI Jiong YU Yuefei WANG Chen BIAN Yonglin PU Yitian ZHANG Yu LIU Load prediction based elastic resource scheduling strategy in Flink Tongxin xuebao stream computing resource scheduling load prediction performance bottleneck Flink |
| title | Load prediction based elastic resource scheduling strategy in Flink |
| title_full | Load prediction based elastic resource scheduling strategy in Flink |
| title_fullStr | Load prediction based elastic resource scheduling strategy in Flink |
| title_full_unstemmed | Load prediction based elastic resource scheduling strategy in Flink |
| title_short | Load prediction based elastic resource scheduling strategy in Flink |
| title_sort | load prediction based elastic resource scheduling strategy in flink |
| topic | stream computing resource scheduling load prediction performance bottleneck Flink |
| url | http://www.joconline.com.cn/zh/article/doi/10.11959/j.issn.1000-436x.2020195/ |
| work_keys_str_mv | AT ziyangli loadpredictionbasedelasticresourceschedulingstrategyinflink AT jiongyu loadpredictionbasedelasticresourceschedulingstrategyinflink AT yuefeiwang loadpredictionbasedelasticresourceschedulingstrategyinflink AT chenbian loadpredictionbasedelasticresourceschedulingstrategyinflink AT yonglinpu loadpredictionbasedelasticresourceschedulingstrategyinflink AT yitianzhang loadpredictionbasedelasticresourceschedulingstrategyinflink AT yuliu loadpredictionbasedelasticresourceschedulingstrategyinflink |