Large-scale log compressing system based on differential compression
The scale of log data produced by the large scale information system is growing rapidly.It leads to the big challenge of line-speed compressing and saving the large scale log data.By analysis on massive network log data,it is found that the log data has redundant pattern in terms of log structure an...
Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | zho |
| Published: |
Editorial Department of Journal on Communications
2015-11-01
|
| Series: | Tongxin xuebao |
| Subjects: | |
| Online Access: | http://www.joconline.com.cn/zh/article/doi/10.11959/j.issn.1000-436x.2015300/ |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1841539582143758336 |
|---|---|
| author | Qiu TANG Lei JIANG Qiong DAI |
| author_facet | Qiu TANG Lei JIANG Qiong DAI |
| author_sort | Qiu TANG |
| collection | DOAJ |
| description | The scale of log data produced by the large scale information system is growing rapidly.It leads to the big challenge of line-speed compressing and saving the large scale log data.By analysis on massive network log data,it is found that the log data has redundant pattern in terms of log structure and time local similarity in terms of log content.A differential log compression architecture based on template is proposed.Fine-grained differential compressive strategies in the architecture can be configured for a special log data.Experimental results show that,compared with gizp,the proposed log compressing architecture improves 2～10 times’ compressive speed and gain a better compressing ratio approaching to 10%. |
| format | Article |
| id | doaj-art-686c5a25f40140f59fc1b90497bbc1c2 |
| institution | Kabale University |
| issn | 1000-436X |
| language | zho |
| publishDate | 2015-11-01 |
| publisher | Editorial Department of Journal on Communications |
| record_format | Article |
| series | Tongxin xuebao |
| spelling | doaj-art-686c5a25f40140f59fc1b90497bbc1c22025-01-14T06:54:26ZzhoEditorial Department of Journal on CommunicationsTongxin xuebao1000-436X2015-11-013619720259697963Large-scale log compressing system based on differential compressionQiu TANGLei JIANGQiong DAIThe scale of log data produced by the large scale information system is growing rapidly.It leads to the big challenge of line-speed compressing and saving the large scale log data.By analysis on massive network log data,it is found that the log data has redundant pattern in terms of log structure and time local similarity in terms of log content.A differential log compression architecture based on template is proposed.Fine-grained differential compressive strategies in the architecture can be configured for a special log data.Experimental results show that,compared with gizp,the proposed log compressing architecture improves 2～10 times’ compressive speed and gain a better compressing ratio approaching to 10%.http://www.joconline.com.cn/zh/article/doi/10.11959/j.issn.1000-436x.2015300/logdifferential compressionfine graintemplate |
| spellingShingle | Qiu TANG Lei JIANG Qiong DAI Large-scale log compressing system based on differential compression Tongxin xuebao log differential compression fine grain template |
| title | Large-scale log compressing system based on differential compression |
| title_full | Large-scale log compressing system based on differential compression |
| title_fullStr | Large-scale log compressing system based on differential compression |
| title_full_unstemmed | Large-scale log compressing system based on differential compression |
| title_short | Large-scale log compressing system based on differential compression |
| title_sort | large scale log compressing system based on differential compression |
| topic | log differential compression fine grain template |
| url | http://www.joconline.com.cn/zh/article/doi/10.11959/j.issn.1000-436x.2015300/ |
| work_keys_str_mv | AT qiutang largescalelogcompressingsystembasedondifferentialcompression AT leijiang largescalelogcompressingsystembasedondifferentialcompression AT qiongdai largescalelogcompressingsystembasedondifferentialcompression |