approved
CORAL: expert-Curated medical Oncology Reports to Advance Language model inference

Both medical care and observational studies in oncology require a thorough understanding of a patient's disease progression and treatment history, often elaborately documented within clinical notes. As large language models (LLMs) are becoming more popular, it becomes important to evaluate their potential in oncology. However, no current information representation schema fully encapsulates the diversity of oncology information within clinical notes, and no comprehensively annotated oncology notes exist publicly, thereby limiting a thorough evaluation. We curated a new fine-grained, expert-labeled dataset of 40 de-identified breast and pancreatic cancer progress notes at University of California, San Francisco, and assessed three recent LLMs (GPT-4, GPT-3.5-turbo, and FLAN-UL2) in zero-shot extraction of detailed oncological information from two narrative sections of clinical progress notes. Model performance was quantified with BLEU-4, ROUGE-1, and exact match (EM) F1-score evaluation metrics. Our team of oncology fellows and medical students manually annotated 9028 entities, 9986 modifiers, and 5312 relationships. The GPT-4 model exhibited overall best performance, with an average BLEU score of 0.73, an average ROUGE score of 0.72, an average EM-F1-score of 0.51, and an average accuracy of 68% (expert manual evaluation on 20 notes). GPT-4 was proficient in tumor characteristics and medication extraction, and demonstrated superior performance in inferring symptoms due to cancer and considerations of future medications. Common errors included partial responses with missing information and hallucinations with note-specific information. LLMs are promising for performing reliable information extraction for clinical research, complex population management, and documenting quality patient care, but there is a need for further improvements.

Tags
Data and Resources
To access the resources you must log in
  • PhysioNet Link

    The resource: 'PhysioNet Link' is not accessible as guest user. You must login to access it!
Personal Data Attributes

Description: Personal Data related Information

Field Value
Anonymised Anonymized
ChildrenData No
General Data Yes
Personal Data Yes
Personal data was manifestly made public by the data subject Yes
Sensitive Data No
Additional Info
Field Value
Accessibility Both
Basic rights Download
Creation Date 2024-02-07 12:15
Creator Sushil, Madhumita
Creator Kennedy, Vanessa
Creator Mandair, Divneet
Creator Miao, Brenda
Creator Zack, Travis
Creator Butte, Atul
Data sharing agreement yes
Dataset Citation Sushil, M., Kennedy, V., Mandair, D., Miao, B., Zack, T., & Butte, A. (2024). CORAL: expert-Curated medical Oncology Reports to Advance Language model inference (version 1.0). PhysioNet. https://doi.org/10.13026/v69y-xa45. Sushil, Madhumita, et al. "Extracting detailed oncologic history and treatment plan from medical oncology notes with large language models." arXiv preprint arXiv:2308.03853 (2023). Goldberger, A., Amaral, L., Glass, L., Hausdorff, J., Ivanov, P. C., Mark, R., ... & Stanley, H. E. (2000). PhysioBank, PhysioToolkit, and PhysioNet: Components of a new research resource for complex physiologic signals. Circulation [Online]. 101 (23), pp. e215–e220.
Field/Scope of use Research only
Group Health Studies
License term 2024-02-07 12:15/2100-02-07 12:15
Processing Degree Secondary
SoBigData Node SoBigData EU
SoBigData Node SoBigData IT
Sublicense rights No
Territory of use World Wide
Thematic Cluster Other
system:type Dataset
Management Info
Field Value
Author d'Aloisio Giordano
Maintainer Patra Payel
Version 1
Last Updated 30 January 2025, 13:58 (CET)
Created 30 January 2025, 12:23 (CET)