-
A Study of Generative Large Language Model for Medical Research and Healthcare
Authors:
Cheng Peng,
Xi Yang,
Aokun Chen,
Kaleb E Smith,
Nima PourNejatian,
Anthony B Costa,
Cheryl Martin,
Mona G Flores,
Ying Zhang,
Tanja Magoc,
Gloria Lipori,
Duane A Mitchell,
Naykky S Ospina,
Mustafa M Ahmed,
William R Hogan,
Elizabeth A Shenkman,
Yi Guo,
Jiang Bian,
Yonghui Wu
Abstract:
There is enormous enthusiasm and concerns in using large language models (LLMs) in healthcare, yet current assumptions are all based on general-purpose LLMs such as ChatGPT. This study develops a clinical generative LLM, GatorTronGPT, using 277 billion words of mixed clinical and English text with a GPT-3 architecture of 20 billion parameters. GatorTronGPT improves biomedical natural language proc…
▽ More
There is enormous enthusiasm and concerns in using large language models (LLMs) in healthcare, yet current assumptions are all based on general-purpose LLMs such as ChatGPT. This study develops a clinical generative LLM, GatorTronGPT, using 277 billion words of mixed clinical and English text with a GPT-3 architecture of 20 billion parameters. GatorTronGPT improves biomedical natural language processing for medical research. Synthetic NLP models trained using GatorTronGPT generated text outperform NLP models trained using real-world clinical text. Physicians Turing test using 1 (worst) to 9 (best) scale shows that there is no significant difference in linguistic readability (p = 0.22; 6.57 of GatorTronGPT compared with 6.93 of human) and clinical relevance (p = 0.91; 7.0 of GatorTronGPT compared with 6.97 of human) and that physicians cannot differentiate them (p < 0.001). This study provides insights on the opportunities and challenges of LLMs for medical research and healthcare.
△ Less
Submitted 22 May, 2023;
originally announced May 2023.
-
GatorTron: A Large Clinical Language Model to Unlock Patient Information from Unstructured Electronic Health Records
Authors:
Xi Yang,
Aokun Chen,
Nima PourNejatian,
Hoo Chang Shin,
Kaleb E Smith,
Christopher Parisien,
Colin Compas,
Cheryl Martin,
Mona G Flores,
Ying Zhang,
Tanja Magoc,
Christopher A Harle,
Gloria Lipori,
Duane A Mitchell,
William R Hogan,
Elizabeth A Shenkman,
Jiang Bian,
Yonghui Wu
Abstract:
There is an increasing interest in developing artificial intelligence (AI) systems to process and interpret electronic health records (EHRs). Natural language processing (NLP) powered by pretrained language models is the key technology for medical AI systems utilizing clinical narratives. However, there are few clinical language models, the largest of which trained in the clinical domain is compar…
▽ More
There is an increasing interest in developing artificial intelligence (AI) systems to process and interpret electronic health records (EHRs). Natural language processing (NLP) powered by pretrained language models is the key technology for medical AI systems utilizing clinical narratives. However, there are few clinical language models, the largest of which trained in the clinical domain is comparatively small at 110 million parameters (compared with billions of parameters in the general domain). It is not clear how large clinical language models with billions of parameters can help medical AI systems utilize unstructured EHRs. In this study, we develop from scratch a large clinical language model - GatorTron - using >90 billion words of text (including >82 billion words of de-identified clinical text) and systematically evaluate it on 5 clinical NLP tasks including clinical concept extraction, medical relation extraction, semantic textual similarity, natural language inference (NLI), and medical question answering (MQA). We examine how (1) scaling up the number of parameters and (2) scaling up the size of the training data could benefit these NLP tasks. GatorTron models scale up the clinical language model from 110 million to 8.9 billion parameters and improve 5 clinical NLP tasks (e.g., 9.6% and 9.5% improvement in accuracy for NLI and MQA), which can be applied to medical AI systems to improve healthcare delivery. The GatorTron models are publicly available at: https://catalog.ngc.nvidia.com/orgs/nvidia/teams/clara/models/gatortron_og.
△ Less
Submitted 16 December, 2022; v1 submitted 2 February, 2022;
originally announced March 2022.
-
Added Value of Intraoperative Data for Predicting Postoperative Complications: Development and Validation of a MySurgeryRisk Extension
Authors:
Shounak Datta,
Tyler J. Loftus,
Matthew M. Ruppert,
Chris Giordano,
Lasith Adhikari,
Ying-Chih Peng,
Yuanfang Ren,
Benjamin Shickel,
Zheng Feng,
Gloria Lipori,
Gilbert R. Upchurch Jr.,
Xiaolin Li,
Parisa Rashidi,
Tezcan Ozrazgat-Baslanti,
Azra Bihorac
Abstract:
To test the hypothesis that accuracy, discrimination, and precision in predicting postoperative complications improve when using both preoperative and intraoperative data input features versus preoperative data alone. Models that predict postoperative complications often ignore important intraoperative physiological changes. Incorporation of intraoperative physiological data may improve model perf…
▽ More
To test the hypothesis that accuracy, discrimination, and precision in predicting postoperative complications improve when using both preoperative and intraoperative data input features versus preoperative data alone. Models that predict postoperative complications often ignore important intraoperative physiological changes. Incorporation of intraoperative physiological data may improve model performance. This retrospective cohort analysis included 52,529 inpatient surgeries at a single institution during a 5 year period. Random forest machine learning models in the validated MySurgeryRisk platform made patient-level predictions for three postoperative complications and mortality during hospital admission using electronic health record data and patient neighborhood characteristics. For each outcome, one model trained with preoperative data alone and one model trained with both preoperative and intraoperative data. Models were compared by accuracy, discrimination (expressed as AUROC), precision (expressed as AUPRC), and reclassification indices (NRI). Machine learning models incorporating both preoperative and intraoperative data had greater accuracy, discrimination, and precision than models using preoperative data alone for predicting all three postoperative complications (intensive care unit length of stay >48 hours, mechanical ventilation >48 hours, and neurological complications including delirium) and in-hospital mortality (accuracy: 88% vs. 77%, AUROC: 0.93 vs. 0.87, AUPRC: 0.21 vs. 0.15). Overall reclassification improvement was 2.9-10.0% for complications and 11.2% for in-hospital mortality. Incorporating both preoperative and intraoperative data significantly increased accuracy, discrimination, and precision for machine learning models predicting postoperative complications.
△ Less
Submitted 8 November, 2019; v1 submitted 28 October, 2019;
originally announced October 2019.
-
Development and validation of computable Phenotype to Identify and Characterize Kidney Health in Adult Hospitalized Patients
Authors:
Tezcan Ozrazgat-Baslanti,
Amir Motaei,
Rubab Islam,
Haleh Hashemighouchani,
Matthew Ruppert,
R. W. M. A. Madushani,
Mark S. Segal,
Gloria Lipori,
Azra Bihorac,
Charles Hobson
Abstract:
Background: Acute kidney injury (AKI) is a common complication in hospitalized patients and a common cause for chronic kidney disease (CKD) and increased hospital cost and mortality. By timely detection of AKI and AKI progression, effective preventive or therapeutic measures could be offered. This study aims to develop and validate an electronic phenotype to identify patients with CKD and AKI. Met…
▽ More
Background: Acute kidney injury (AKI) is a common complication in hospitalized patients and a common cause for chronic kidney disease (CKD) and increased hospital cost and mortality. By timely detection of AKI and AKI progression, effective preventive or therapeutic measures could be offered. This study aims to develop and validate an electronic phenotype to identify patients with CKD and AKI. Methods: A database with electronic health records data from a retrospective study cohort of 84,352 hospitalized adults was created. This repository includes demographics, comorbidities, vital signs, laboratory values, medications, diagnoses and procedure codes for all index admission, 12 months prior and 12 months follow-up encounters. We developed algorithms to identify CKD and AKI based on the Kidney Disease: Improving Global Outcomes (KDIGO) criteria. To measure diagnostic performance of the algorithms, clinician experts performed clinical adjudication of AKI and CKD on 300 selected cases. Results: Among 149,136 encounters, identified CKD by medical history was 12% which increased to 16% using creatinine criteria. Among 130,081 encounters with sufficient data for AKI phenotyping 21% had AKI. The comparison of CKD phenotyping algorithm to manual chart review yielded PPV of 0.87, NPV of 0.99, sensitivity of 0.99, and specificity of 0.89. The comparison of AKI phenotyping algorithm to manual chart review yielded PPV of 0.99, NPV of 0.95 , sensitivity 0.98, and specificity 0.98. Conclusions: We developed phenotyping algorithms that yielded very good performance in identification of patients with CKD and AKI in validation cohort. This tool may be useful in identifying patients with kidney disease in a large population, in assessing the quality and value of care in such patients.
△ Less
Submitted 26 March, 2019; v1 submitted 7 March, 2019;
originally announced March 2019.
