REVIEW ARTICLE
Quality assessment of ChatGPT-generated medical information for healthcare professionals and patients: a systematic reviewv
1
Doctoral School, Collegium Medicum, Jan Kochanowski University, Kielce, Poland
2
Department of Education and Research in Health Sciences, Faculty of Health Sciences, Medical University of Warsaw, Poland
3
Collegium Medicum, Jan Kochanowski University, Kielce, Poland
These authors had equal contribution to this work
Submission date: 2025-07-10
Final revision date: 2025-08-03
Acceptance date: 2025-08-04
Online publication date: 2026-09-18
Corresponding author
KEYWORDS
TOPICS
ABSTRACT
Introduction:
ChatGPT, an artificial intelligence (AI)-based language model, has gained popularity in medicine, although concerns remain regarding the accuracy and clinical applicability of its generated content.
Methods:
In accordance with PRISMA guidelines, a systematic review was conducted using the PubMed, EBSCO, and Scopus databases, including 29 studies published between 2022 and 2024. The review included publications evaluating the accuracy, readability, comprehensiveness, and clinical utility of medical information generated by ChatGPT.
Results:
ChatGPT’s content was generally consistent with clinical knowledge, although the quality varied depending on language, query formulation, and medical domain. Responses in English were usually more accurate. Limitations included lack of references, insufficient contextual and emotional understanding, and inconsistencies with medical guidelines in complex cases.
Conclusions:
ChatGPT shows potential as an educational and clinical decision-support tool but requires further refinement and standardisation of evaluation methods.
ChatGPT, an artificial intelligence (AI)-based language model, has gained popularity in medicine, although concerns remain regarding the accuracy and clinical applicability of its generated content.
Methods:
In accordance with PRISMA guidelines, a systematic review was conducted using the PubMed, EBSCO, and Scopus databases, including 29 studies published between 2022 and 2024. The review included publications evaluating the accuracy, readability, comprehensiveness, and clinical utility of medical information generated by ChatGPT.
Results:
ChatGPT’s content was generally consistent with clinical knowledge, although the quality varied depending on language, query formulation, and medical domain. Responses in English were usually more accurate. Limitations included lack of references, insufficient contextual and emotional understanding, and inconsistencies with medical guidelines in complex cases.
Conclusions:
ChatGPT shows potential as an educational and clinical decision-support tool but requires further refinement and standardisation of evaluation methods.
REFERENCES (41)
1.
Sarkar C, Das B, Rawat VS, Wahlang JB, Nongpiur A, Tiewsoh I, Lyngdoh NM, Das D, Bidarolli M, Sony HT. Artificial intelligence and machine learning technology driven modern drug discovery and development. Int J Mol Sci. 2023; 24(3): 2026.
2.
Ossowska A, Kusiak A, Świetlik D. Artificial intelligence in dentistry-narrative review. Int J Environ Res Public Health. 2022; 19(6): 3449.
3.
Nowy Marketing. Raport Digital 2024 Global Overview: Jaki był 2023 rok w Internecie i mediach społecznościowych [Internet]. 2024 [cytowane 2024 Lip 23]. Available at: https://nowymarketing.pl/rapor....
4.
Kokka I, Mourikis I, Nicolaides NC, Darviri C, Chrousos GP, Kanaka-Gantenbein C, Bacopoulou F. Exploring the effects of problematic internet use on adolescent sleep: a systematic review. Int J Environ Res Public Health. 2021; 18(2): 760.
5.
Biswas S, Dobaria D, Cohen HL. ChatGPT and the future of journal reviews: a feasibility study. Yale J Biol Med. 2023; 96(3): 415-420.
6.
Sallam M. ChatGPT utility in healthcare education, research, and practice: systematic review on the promising perspectives and valid concerns. Healthcare (Basel). 2023; 11(6): 887.
7.
Liu J, Wang C, Liu S. Utility of ChatGPT in clinical practice. J Med Internet Res. 2023; 25: e48568.
8.
Dergaa I, Chamari K, Zmijewski P, Saad HB. From human writing to artificial intelligence generated text: examining the prospects and potential threats of ChatGPT in academic writing. Biol Sport. 2023; 40(2): 615-622.
9.
Gödde D, Nöhl S, Wolf C, Rupert Y, Rimkus L, Ehlers J, Breuckmann F, Sellmann T. A SWOT (strengths, weaknesses, opportunities, and threats) analysis of ChatGPT in the medical literature: concise review. J Med Internet Res. 2023; 25: e49368.
10.
Moher D, Liberati A, Tetzlaff J, Altman DG; PRISMA Group. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS Med. 2009; 6(7): e1000097.
11.
Ibrahim MT, Khaskheli SA, Shahzad H, Noordin S. Language-adaptive artificial intelligence: assessing ChatGPT’s answer to frequently asked questions on total hip arthroplasty questions. J Pak Med Assoc. 2024; (Suppl. 4): 74.
12.
Sallam M, Salim NA, Barakat M, Al-Tammemi AB. ChatGPT applications in medical, dental, pharmacy, and public health education: a descriptive study highlighting the advantages and limitations. Narra J. 2023; 3(1): e103.
13.
Hernandez CA, Vazquez Gonzalez AE, Polianovskaia A, Sanchez RA, Arce VM, Mustafa A, Vypritskaya E, Gutierrez OP, Bashir M, Sedeh AE. The future of patient education: AI-driven guide for type 2 diabetes. Cureus. 2023; 15(11): e48919.
14.
Alanzi TM, Alzahrani W, Albalawi NS, Allahyani T, Alghamdi A, Al-Zahrani H, Almutairi A, Alzahrani H, Almulhem L, Alanzi N, Al Moarfeg A, Farhah N. Public awareness of obesity as a risk factor for cancer in Central Saudi Arabia: feasibility of ChatGPT as an educational intervention. Cureus. 2023; 15(12): e50781.
15.
Aliyeva A, Sari E, Alaskarov E, Nasirov R. Enhancing postoperative cochlear implant care with ChatGPT-4: a study on artificial intelligence (AI)-assisted patient education and support. Cureus. 2024; 16(2): e53897.
16.
Kasapovic A, Ali T, Babasiz M, Bojko J, Gathen M, Kaczmarczyk R, Roos J. Does the information quality of ChatGPT meet the requirements of orthopedics and trauma surgery? Cureus. 2024; 16(5): e60318.
17.
Anastasio AT, Mills IV FB, Karavan Jr MP, Adams Jr SB. Evaluating the quality and usability of artificial intelligence–generated responses to common patient questions in foot and ankle surgery. Foot Ankle Orthop. 2023; 8(4): 24730114231209919.
18.
King RC, Samaan JS, Yeo YH, Peng Y, Kunkel DC, Habib AA, Ghashghaei R. A multidisciplinary assessment of ChatGPT’s knowledge of amyloidosis: observational study. JMIR Cardio. 2024; 8: e53421.
19.
Giorgino R, Alessandri-Bonetti M, Del Re M. Evaluating ChatGPT and Google Bard as educational tools for patients in orthopedics. Diagnostics. 2024; 14: 1253.
20.
Dergaa I, Fekih-Romdhane F, Hallit S, Loch AA, Glenn JM, Fessi MS, Aissa MB, Souissi N, Guelmami N, Swed S, El Omri A, Bragazzi NL, Saad HB. ChatGPT is not ready yet for use in providing mental health assessment and interventions. Front Psychiatry. 2024; 14: 1277756.
21.
Sahin S, Erkmen B, Duymaz YK, Bayram F, Tekin AM, Topsakal V. Evaluating ChatGPT-4’s performance as a digital health advisor for otosclerosis surgery. Front Surg. 2024; 11: 1373843.
22.
Shahsavar Y, Choudhury A. User intentions to use ChatGPT for self-diagnosis and health-related purposes: cross-sectional survey study. JMIR Hum Factors. 2023; 10: e47564.
23.
Xue E, Bracken-Clarke D, Iannantuono GM, Choo-Wosoba H, Gulley JL, Floudas CS. Utility of large language models for health care professionals and patients in navigating hematopoietic stem cell transplantation: comparison of the performance of ChatGPT-3.5, ChatGPT-4, and Bard. J Med Internet Res. 2024; 26: e54758.
24.
Lv X, Zhang X, Li Y, Ding X, Lai H, Shi J. Leveraging large language models for improved patient access and self-management: assessor-blinded comparison between expert- and AI-generated content. J Med Internet Res. 2024; 26: e55847.
25.
Fahy S, Niemann M, Böhm P, Winkler T, Oehme S. Assessment of the quality and readability of information provided by ChatGPT in relation to the use of platelet-rich plasma therapy for osteoarthritis. J Pers Med. 2024; 14: 495.
26.
Gray M, Baird A, Sawyer T, James J, DeBroux T, Bartlett M, Krick J, Umoren R. Increasing realism and variety of virtual patient dialogues for prenatal counseling education through a novel application of ChatGPT: exploratory observational study. JMIR Med Educ. 2024; 10: e50705.
27.
Sivarajkumar S, Gao F, Denny P, Aldhahwani B, Visweswaran S, Bove A, Wang Y. Mining clinical notes for physical rehabilitation exercise information: natural language processing algorithm development and validation study. JMIR Med Inform. 2024; 12: e52289.
28.
Liao LL, Chang LC, Lai IJ. Assessing the quality of Chat- GPT’s dietary advice for college students from dietitians’ perspectives. Nutrients. 2024; 16: 1939.
29.
Dennstädt F, Hastings J, Putora PM, Vu E, Fischer GF, Süveg K, Glatzer M, Riggenbach E, Hà HL, Cihoric N. Exploring the capabilities of large language models in answering domain-specific questions related to radiation oncology. Adv Radiat Oncol. 2024; 9: 101400.
30.
Thia I, Saluja M. ChatGPT: is this patient education tool for urological malignancies readable for the general population? Res Rep Urol. 2024; 16: 31-37.
31.
Walker HL, Ghani S, Kuemmerli C, Nebiker CA, Müller BP, Raptis DA, Staubli SM. Reliability of medical information provided by ChatGPT: assessment against clinical guidelines and patient information quality instrument. J Med Internet Res. 2023; 25: e47479.
32.
Mondal H, Dash I, Mondal S, Behera JK. ChatGPT in answering queries related to lifestyle-related diseases and disorders. Cureus. 2023; 15: e48296.
33.
Yeo YH, Samaan JS, Ng WH, Ting PS, Trivedi H, Vipani A, Ayoub W, Yang JD, Liran O, Spiegel B, Kuo A. Assessing the performance of ChatGPT in answering questions regarding cirrhosis and hepatocellular carcinoma. Clin Mol Hepatol. 2023; 29(3): 721-732.
34.
Alan R, Alan BM. Utilizing ChatGPT-4 for providing information on periodontal disease to patients: a DISCERN quality analysis. Cureus. 2023; 15: e46213.
35.
Haver HL, Gupta AK, Ambinder EB, Bahl M, Oluyemi ET, Jeudy J, Yi PH. Appropriateness of breast cancer prevention and screening recommendations provided by ChatGPT. Radiology. 2023; 307(4): e230424.
36.
Rahsepar AA, Tavakoli N, Kim GHJ, Hassani C, Abtin F, Bedayat A. How AI responds to common lung cancer questions: ChatGPT vs Google Bard. Radiology. 2023; 307(5): e230922.
37.
Seth I, Lim B, Xie Y, Cevik J, Rozen WM, Ross RJ, Lee M. Comparing the efficacy of large language models ChatGPT, BARD, and Bing AI in providing information on rhinoplasty: an observational study. Aesthet Surg J Open Forum. 2023; 5: ojad084.
38.
Mondal H, Mondal S, Podder I. Using ChatGPT for writing articles for patients’ education for dermatological diseases: a pilot study. Indian Dermatol Online J. 2023; 14: 482-486.
39.
Ponzo V, Goitre I, Favaro E, Merio FD, Mancino MV, Riso S, Bo S. Is ChatGPT an effective tool for providing dietary advice? Nutrients. 2024; 16: 469.
40.
Drabiak K. Leveraging law and ethics to promote safe and reliable AI/ML in healthcare. Front Nucl Med. 2022; 2: 983340.
41.
Baumgartner R, Arora P, Bath C, Burljaev D, Ciereszko K, Custers B, Ding J, Ernst W, Fosch-Villaronga E, Galanos V, Gremsl T, Hendl T, Kropp C, Lenk C, Martin P, Mbelu S, Dos Santos Bruss SM, Napiwodzka K, Nowak E, Roxan- ne T, Samerski S, Schneeberger D, Tampe-Mai K, Vlanto- ni K, Wiggert K, Williams R. Fair and equitable AI in biomedical research and healthcare: social science perspectives. Artif Intell Med. 2023; 144: 102658.
Share
RELATED ARTICLE
| eISSN: | 2300-6722 |
| ISSN: | 1899-1874 |
We process personal data collected when visiting the website. The function of obtaining information about users and their behavior is carried out by voluntarily entered information in forms and saving cookies in end devices. Data, including cookies, are used to provide services, improve the user experience and to analyze the traffic in accordance with the Privacy policy. Data are also collected and processed by Google Analytics tool (more).
You can change cookies settings in your browser. Restricted use of cookies in the browser configuration may affect some functionalities of the website.
You can change cookies settings in your browser. Restricted use of cookies in the browser configuration may affect some functionalities of the website.
