Research and implementation on acquisition scheme of telecom big data based on Hadoop
ETL is a very important step in the implementation process of data warehouse.A good ETL flow is important,which can effectively process the telecom big data and improve the acquisition efficiency of the operation platform.Firstly,the main data content of the big data platform was expounded.Secondly,...
Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | zho |
| Published: |
Beijing Xintong Media Co., Ltd
2017-01-01
|
| Series: | Dianxin kexue |
| Subjects: | |
| Online Access: | http://www.telecomsci.com/zh/article/doi/10.11959/j.issn.1000-0801.2017010/ |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1841530084457971712 |
|---|---|
| author | Baoyou WANG Jing QIAN Shijin YUAN |
| author_facet | Baoyou WANG Jing QIAN Shijin YUAN |
| author_sort | Baoyou WANG |
| collection | DOAJ |
| description | ETL is a very important step in the implementation process of data warehouse.A good ETL flow is important,which can effectively process the telecom big data and improve the acquisition efficiency of the operation platform.Firstly,the main data content of the big data platform was expounded.Secondly,in order to improve the efficiency of massive data collection,Hadoop and Oracle mashup solution was suggested.Subsequently,a dynamic triggered ETL scheduling flow and algorithm was proposed.Compared with timer start ETL scheduling method,it could effectively shorten waiting time and avoid the phenomenon of resources to seize and congestion.Finally,according to the running log of Hadoop platform and Oracle database,the relationship between acquisition efficiency and data quantity was analyzed comparatively.Furthermore,practice result shows that the hybrid data structure of the big data platform complement each other and can effectively enhance the timeliness of data collection and access better application effect. |
| format | Article |
| id | doaj-art-c586993b9fde4e569f4be62ea44905f5 |
| institution | Kabale University |
| issn | 1000-0801 |
| language | zho |
| publishDate | 2017-01-01 |
| publisher | Beijing Xintong Media Co., Ltd |
| record_format | Article |
| series | Dianxin kexue |
| spelling | doaj-art-c586993b9fde4e569f4be62ea44905f52025-01-15T03:13:34ZzhoBeijing Xintong Media Co., LtdDianxin kexue1000-08012017-01-013313514259604166Research and implementation on acquisition scheme of telecom big data based on HadoopBaoyou WANGJing QIANShijin YUANETL is a very important step in the implementation process of data warehouse.A good ETL flow is important,which can effectively process the telecom big data and improve the acquisition efficiency of the operation platform.Firstly,the main data content of the big data platform was expounded.Secondly,in order to improve the efficiency of massive data collection,Hadoop and Oracle mashup solution was suggested.Subsequently,a dynamic triggered ETL scheduling flow and algorithm was proposed.Compared with timer start ETL scheduling method,it could effectively shorten waiting time and avoid the phenomenon of resources to seize and congestion.Finally,according to the running log of Hadoop platform and Oracle database,the relationship between acquisition efficiency and data quantity was analyzed comparatively.Furthermore,practice result shows that the hybrid data structure of the big data platform complement each other and can effectively enhance the timeliness of data collection and access better application effect.http://www.telecomsci.com/zh/article/doi/10.11959/j.issn.1000-0801.2017010/big dataETLHadoopscheduling processmashup architecture |
| spellingShingle | Baoyou WANG Jing QIAN Shijin YUAN Research and implementation on acquisition scheme of telecom big data based on Hadoop Dianxin kexue big data ETL Hadoop scheduling process mashup architecture |
| title | Research and implementation on acquisition scheme of telecom big data based on Hadoop |
| title_full | Research and implementation on acquisition scheme of telecom big data based on Hadoop |
| title_fullStr | Research and implementation on acquisition scheme of telecom big data based on Hadoop |
| title_full_unstemmed | Research and implementation on acquisition scheme of telecom big data based on Hadoop |
| title_short | Research and implementation on acquisition scheme of telecom big data based on Hadoop |
| title_sort | research and implementation on acquisition scheme of telecom big data based on hadoop |
| topic | big data ETL Hadoop scheduling process mashup architecture |
| url | http://www.telecomsci.com/zh/article/doi/10.11959/j.issn.1000-0801.2017010/ |
| work_keys_str_mv | AT baoyouwang researchandimplementationonacquisitionschemeoftelecombigdatabasedonhadoop AT jingqian researchandimplementationonacquisitionschemeoftelecombigdatabasedonhadoop AT shijinyuan researchandimplementationonacquisitionschemeoftelecombigdatabasedonhadoop |