PharmaLLM: A Medicine Prescriber Chatbot Exploiting Open-Source Large Language Models
Abstract The increasing adoption of large language models (LLMs) in healthcare presents both opportunities and challenges. While LLM-powered applications are being utilized for various medical tasks, concerns persist regarding their accuracy and reliability, particularly when not specifically traine...
Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Springer Nature
2024-11-01
|
| Series: | Human-Centric Intelligent Systems |
| Subjects: | |
| Online Access: | https://doi.org/10.1007/s44230-024-00085-z |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1841544575303286784 |
|---|---|
| author | Ayesha Azam Zubaira Naz Muhammad Usman Ghani Khan |
| author_facet | Ayesha Azam Zubaira Naz Muhammad Usman Ghani Khan |
| author_sort | Ayesha Azam |
| collection | DOAJ |
| description | Abstract The increasing adoption of large language models (LLMs) in healthcare presents both opportunities and challenges. While LLM-powered applications are being utilized for various medical tasks, concerns persist regarding their accuracy and reliability, particularly when not specifically trained on medical data. Using open-source models without proper fine-tuning for medical applications can lead to inaccurate or potentially harmful advice, underscoring the need for domain-specific adaptation. Therefore, this study addresses these issues by developing PharmaLLM, a fine-tuned version of the open-source Llama 2 model, designed to provide accurate medicine prescription information. PharmaLLM incorporates a multi-modal input/output mechanism, supporting both text and speech, to enhance accessibility. The fine-tuning process utilized LoRA (Low-Rank Adaptation) with a rank of 16 for parameter-efficient fine-tuning. The learning rate was maintained at 2e-4 for stable adjustments, and a batch size of 12 was chosen to balance computational efficiency and learning effectiveness. The system demonstrated strong performance metrics, achieving 87% accuracy, 92.16% F1 score, 94% sensitivity, 66% specificity, and 90% precision. A usability study involving 33 participants was conducted to evaluate the system using the Chatbot Usability Questionnaire, focusing on error handling, response generation, navigation, and personality. Results from the questionnaire indicated that participants found the system easy to navigate and the responses useful and relevant. PharmaLLM aims to facilitate improved patient-physician interactions, particularly in areas with limited healthcare resources and low literacy rates. This research contributes to the advancement of medical informatics by offering a reliable, accessible web-based tool that benefits both patients and healthcare providers. |
| format | Article |
| id | doaj-art-36432b17812f48c981e5f27890def03e |
| institution | Kabale University |
| issn | 2667-1336 |
| language | English |
| publishDate | 2024-11-01 |
| publisher | Springer Nature |
| record_format | Article |
| series | Human-Centric Intelligent Systems |
| spelling | doaj-art-36432b17812f48c981e5f27890def03e2025-01-12T12:26:38ZengSpringer NatureHuman-Centric Intelligent Systems2667-13362024-11-014452754410.1007/s44230-024-00085-zPharmaLLM: A Medicine Prescriber Chatbot Exploiting Open-Source Large Language ModelsAyesha Azam0Zubaira Naz1Muhammad Usman Ghani Khan2Department of Computer Science, University of Engineering and TechnologyAl-Khwarizmi Institute of Computer Science, University of Engineering and TechnologyDepartment of Computer Science, University of Engineering and TechnologyAbstract The increasing adoption of large language models (LLMs) in healthcare presents both opportunities and challenges. While LLM-powered applications are being utilized for various medical tasks, concerns persist regarding their accuracy and reliability, particularly when not specifically trained on medical data. Using open-source models without proper fine-tuning for medical applications can lead to inaccurate or potentially harmful advice, underscoring the need for domain-specific adaptation. Therefore, this study addresses these issues by developing PharmaLLM, a fine-tuned version of the open-source Llama 2 model, designed to provide accurate medicine prescription information. PharmaLLM incorporates a multi-modal input/output mechanism, supporting both text and speech, to enhance accessibility. The fine-tuning process utilized LoRA (Low-Rank Adaptation) with a rank of 16 for parameter-efficient fine-tuning. The learning rate was maintained at 2e-4 for stable adjustments, and a batch size of 12 was chosen to balance computational efficiency and learning effectiveness. The system demonstrated strong performance metrics, achieving 87% accuracy, 92.16% F1 score, 94% sensitivity, 66% specificity, and 90% precision. A usability study involving 33 participants was conducted to evaluate the system using the Chatbot Usability Questionnaire, focusing on error handling, response generation, navigation, and personality. Results from the questionnaire indicated that participants found the system easy to navigate and the responses useful and relevant. PharmaLLM aims to facilitate improved patient-physician interactions, particularly in areas with limited healthcare resources and low literacy rates. This research contributes to the advancement of medical informatics by offering a reliable, accessible web-based tool that benefits both patients and healthcare providers.https://doi.org/10.1007/s44230-024-00085-zChatbotsFine tuningGenerative AILarge language modelsMedicine |
| spellingShingle | Ayesha Azam Zubaira Naz Muhammad Usman Ghani Khan PharmaLLM: A Medicine Prescriber Chatbot Exploiting Open-Source Large Language Models Human-Centric Intelligent Systems Chatbots Fine tuning Generative AI Large language models Medicine |
| title | PharmaLLM: A Medicine Prescriber Chatbot Exploiting Open-Source Large Language Models |
| title_full | PharmaLLM: A Medicine Prescriber Chatbot Exploiting Open-Source Large Language Models |
| title_fullStr | PharmaLLM: A Medicine Prescriber Chatbot Exploiting Open-Source Large Language Models |
| title_full_unstemmed | PharmaLLM: A Medicine Prescriber Chatbot Exploiting Open-Source Large Language Models |
| title_short | PharmaLLM: A Medicine Prescriber Chatbot Exploiting Open-Source Large Language Models |
| title_sort | pharmallm a medicine prescriber chatbot exploiting open source large language models |
| topic | Chatbots Fine tuning Generative AI Large language models Medicine |
| url | https://doi.org/10.1007/s44230-024-00085-z |
| work_keys_str_mv | AT ayeshaazam pharmallmamedicineprescriberchatbotexploitingopensourcelargelanguagemodels AT zubairanaz pharmallmamedicineprescriberchatbotexploitingopensourcelargelanguagemodels AT muhammadusmanghanikhan pharmallmamedicineprescriberchatbotexploitingopensourcelargelanguagemodels |