Research and application of defense mechanism for prompt injection attack of large language model in financial industry
The large language models had a broad application prospect in the financial sector, and they were expected to play an important role in both asset management and wealth management. With the rapid development and wide application of large language models such as ChatGPT and GPT-4, attention to the se...
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
POSTS&TELECOM PRESS Co., LTD
2024-10-01
|
| Series: | 网络与信息安全学报 |
| Subjects: | |
| Online Access: | http://www.cjnis.com.cn/thesisDetails#10.11959/j.issn.2096-109x.2024071 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1841529529049284608 |
|---|---|
| author | MOU Daen WEI Zhihua SUN Minglong SONG Na NI Lin |
| author_facet | MOU Daen WEI Zhihua SUN Minglong SONG Na NI Lin |
| author_sort | MOU Daen |
| collection | DOAJ |
| description | The large language models had a broad application prospect in the financial sector, and they were expected to play an important role in both asset management and wealth management. With the rapid development and wide application of large language models such as ChatGPT and GPT-4, attention to the security of large language models increased. The financial industry, characterized by strict regulations and supervision, demanded heightened security measures. Consequently, a comprehensive study on prompt injection attacks and a security defense framework was delved into in large language models within the financial sector. A risk taxonomy encompassing eight forms of input prompt injection attacks and five categories of safety scenarios on the output side was developed, and a financial domain large model prompt injection attack dataset, FIN-CSAPrompts, was collected using a human-machine adversarial approach. An end-to-end security defense framework against prompt injection attacks in large language models was proposed and tested, and comparative evaluations were performed using prevalent open-source large language models. The research indicated that in the financial industry, the application of the proposed security defense framework significantly enhanced the defensive capabilities of Chinese large language models, effectively reducing the generation of inappropriate content and improving their resilience against prompt injection attacks. This research provided a reference and foundation for further research on the security issues of Chinese large language models in the financial domain, offering datasets, evaluation metrics, and solutions for consideration and adaptation. |
| format | Article |
| id | doaj-art-6f136b7dace24418aff410bec522d874 |
| institution | Kabale University |
| issn | 2096-109X |
| language | English |
| publishDate | 2024-10-01 |
| publisher | POSTS&TELECOM PRESS Co., LTD |
| record_format | Article |
| series | 网络与信息安全学报 |
| spelling | doaj-art-6f136b7dace24418aff410bec522d8742025-01-15T03:17:21ZengPOSTS&TELECOM PRESS Co., LTD网络与信息安全学报2096-109X2024-10-011011913377772303Research and application of defense mechanism for prompt injection attack of large language model in financial industryMOU DaenWEI ZhihuaSUN MinglongSONG NaNI LinThe large language models had a broad application prospect in the financial sector, and they were expected to play an important role in both asset management and wealth management. With the rapid development and wide application of large language models such as ChatGPT and GPT-4, attention to the security of large language models increased. The financial industry, characterized by strict regulations and supervision, demanded heightened security measures. Consequently, a comprehensive study on prompt injection attacks and a security defense framework was delved into in large language models within the financial sector. A risk taxonomy encompassing eight forms of input prompt injection attacks and five categories of safety scenarios on the output side was developed, and a financial domain large model prompt injection attack dataset, FIN-CSAPrompts, was collected using a human-machine adversarial approach. An end-to-end security defense framework against prompt injection attacks in large language models was proposed and tested, and comparative evaluations were performed using prevalent open-source large language models. The research indicated that in the financial industry, the application of the proposed security defense framework significantly enhanced the defensive capabilities of Chinese large language models, effectively reducing the generation of inappropriate content and improving their resilience against prompt injection attacks. This research provided a reference and foundation for further research on the security issues of Chinese large language models in the financial domain, offering datasets, evaluation metrics, and solutions for consideration and adaptation.http://www.cjnis.com.cn/thesisDetails#10.11959/j.issn.2096-109x.2024071financial large language model securityprompt injectionrisk taxonomylarge model datasetillegal risk detection |
| spellingShingle | MOU Daen WEI Zhihua SUN Minglong SONG Na NI Lin Research and application of defense mechanism for prompt injection attack of large language model in financial industry 网络与信息安全学报 financial large language model security prompt injection risk taxonomy large model dataset illegal risk detection |
| title | Research and application of defense mechanism for prompt injection attack of large language model in financial industry |
| title_full | Research and application of defense mechanism for prompt injection attack of large language model in financial industry |
| title_fullStr | Research and application of defense mechanism for prompt injection attack of large language model in financial industry |
| title_full_unstemmed | Research and application of defense mechanism for prompt injection attack of large language model in financial industry |
| title_short | Research and application of defense mechanism for prompt injection attack of large language model in financial industry |
| title_sort | research and application of defense mechanism for prompt injection attack of large language model in financial industry |
| topic | financial large language model security prompt injection risk taxonomy large model dataset illegal risk detection |
| url | http://www.cjnis.com.cn/thesisDetails#10.11959/j.issn.2096-109x.2024071 |
| work_keys_str_mv | AT moudaen researchandapplicationofdefensemechanismforpromptinjectionattackoflargelanguagemodelinfinancialindustry AT weizhihua researchandapplicationofdefensemechanismforpromptinjectionattackoflargelanguagemodelinfinancialindustry AT sunminglong researchandapplicationofdefensemechanismforpromptinjectionattackoflargelanguagemodelinfinancialindustry AT songna researchandapplicationofdefensemechanismforpromptinjectionattackoflargelanguagemodelinfinancialindustry AT nilin researchandapplicationofdefensemechanismforpromptinjectionattackoflargelanguagemodelinfinancialindustry |