Ishan Perera

Abstract

Traditionally, medical students have been evaluated on their performance and reporting of history and physical exam i n Standardized Patient experiences within the Hospital Integrated Clinical Cases (HICC) course to determine their mastery over the subject material. At the Edward Via College of Osteopathic Medicine (VCOM) students have always been evaluated on both their performance within the simulated patient encounter and their ability to properly document their Medical Decision Making (MDM). Recently, in accordance with the January 1st, 2023, shift in CPT Evaluation and Management guidelines to focus on MDM and visit timing rather than history and physical exam, and the goal to produce medical students with dense medical lexicons, the VCOM Virginia Campus has begun evaluating students according to a pre-determined set of keywords, as established for their specific Standardized Patient case. (AMA) Our proprietary MDM Keyword Analyzer is able to accurately convert audio-based HPI presentations into textual input to analyze and provide feedback to students on their mastery of the medical lexicon and to determine critical criteria for MDM, such as New York Heart Association (NYHA) Class for standardized classification of heart failure symptoms. (Daniels et al., 2023) In continuation of this previous project and evolution of the concept, this study will discuss the use of a Voice Recognition-based System paired to the Random Forests analytical method. This system will be evaluated on its ability to assess medical student’s MDM based on pre-determined keywords in the setting of high fidelity medical simulations or Standardized Patient encounters while recognizing the many challenges to voice recognition, such as converted audio that is not clear or assessing an unstructured recording. (Quiroz et al., 2019) Specifically, recent analysis has found modern voice recognition models used in MDM carry a 7% error rate – meaning 7 out of every 100 auto-transcribed words contain a transcription error. (Zhou et al., 2018) This study’s purpose is to evaluate a Voice Recognition-based proprietary technique in its ability to assess medical student’s clinical decision making based on pre-determined keywords.

References

1. American Medical Association. CPT® Evaluation and Management. American Medical Association.

https://www.ama-assn.org/practice-management/cpt/cpt-evaluation-and-management. Published December 27, 2023. Accessed January 19, 2024. 2. Commission on Osteopathic College Accreditation. Accreditation of Colleges of Osteopathic Medicine: COM Continuing Accreditation Standards. American Osteopathic Association; 2023.

https://osteopathic.org/index.php?aam-media=/wp-content/uploads/COCA-2023-COM-Continuing-Standards.pdf. 3. Goss FR, Blackley SV, Ortega CA, et al. A clinician survey of using speech recognition for clinical documentation in the electronic health record. Int J Med Inform. 2019;130:103938.

4. Kauppinen T, Koivikko MP, Ahovuo J. Improvement of report workflow and productivity using speech recognition--a follow-up study. J Digit Imaging. 2008;21(4):378-382.

5. Radford, A., Kim, J. W., Xu, T., Brockman, G., McLeavey, C., & Sutskever, I. (2022). Robust Speech Recognition via Large-Scale Weak Supervision. ms, San Fransisco. 

6. Introducing whisper. Introducing Whisper. (n.d.). https://openai.com/research/whisper