Abstract
We assessed the usability of consultation order templates and identified problems to prioritize in design efforts for improving referral communication. With a sample of 26 consultation order templates, three evaluators performed a usability heuristic evaluation. The evaluation used 14 domain-independent heuristics and the following three supplemental references: 1 new domain-specific heuristic, 6 usability goals, and coded clinicians’ statements regarding ease of use for 10 sampled templates. Evaluators found 201 violations, a mean of 7.7 violations per template. Minor violations outnumbered major violations almost twofold, 115 (57%) to 62 (31%). Approximately 68% of violations were linked to 5 heuristics: aesthetic and minimalist design (17%), error prevention (16%), consistency and standards (14%), recognition rather than recall (11%), and meet referrers’ information needs (10%). Severe violations were attributed mostly to meet referrers’ information needs and recognition rather than recall. Recorded violations yielded potential negative consequences for efficiency, effectiveness, safety, learnability, and utility. Evaluators and clinicians demonstrated 80% agreement in usability assessment. Based on frequency and severity of usability heuristic violations, the consultation order templates reviewed may impede clinical efficiency and risk patient safety. Results support the following design considerations: communicate consultants’ requirements, facilitate information seeking, and support communication. While the most frequent heuristic violations involved interaction design and presentation, the most severe violations lacked information desired by referring clinicians. Violations related to templates’ inability to support referring clinicians’ information needs had the greatest potential negative impact on efficiency and safety usability goals. Heuristics should be prioritized in future design efforts.
Similar content being viewed by others
References
Kirsh, S.R., Ho, P.M., and Aron, D.C., Providing specialty consultant expertise to primary care: An expanding spectrum of modalities. Mayo Clin Proc. 89(10):1416(1411), 2014. doi:10.1016/j.mayocp.2014.04.016.
Barnett, M.L., Song, Z., and Landon, B.E., Trends in physician referrals in the United States, 1999-2009. Arch Intern Med. 172(2):163–170, 2012. doi:10.1001/archinternmed.2011.722.
Mehrotra, A., Forrest, C.B., and Lin, C.Y., Dropping the baton: Specialty referrals in the United States. Milbank Q. 89(1):39–68, 2011. doi:10.1111/j.1468-0009.2011.00619.x.
Zuchowski, J.L., Rose, D.E., Hamilton, A.B., Stockdale, S.E., Meredith, L.S., Yano, E.M., Rubenstein, L.V., and Cordasco, K.M., Challenges in referral communication between VHA primary care and specialty care. J Gen Intern Med. 30(3):305–311, 2015. doi:10.1007/s11606-014-3100-x.
O’Malley, A.S., and Reschovsky, J.D., Referral and consultation communication between primary care and specialist physicians: Finding common ground. Arch Intern Med. 171(1):56–65, 2011. doi:10.1001/archinternmed.2010.480.
Conley, J., Jordan, M., and Ghali, W.A., Audit of the consultation process on general internal medicine services. Quality & Safety in Health Care. 18(1):59–62, 2009. doi:10.1136/qshc.2007.025486.
Salerno, S.M., Hurst, F.P., Halvorson, S., and Mercado, D.L., Principles of effective consultation: An update for the 21st-century consultant. Arch Intern Med. 167(3):271–275, 2007. doi:10.1001/archinte.167.3.271.
Saleem, J.J., Russ, A.L., Neddo, A., Blades, P.T., Doebbeling, B.N., and Foresman, B.H., Paper persistence, workarounds, and communication breakdowns in computerized consultation management. Int J Med Inform. 80(7):466–479, 2011. doi:10.1016/j.ijmedinf.2011.03.016.
Hysong, S.J., Esquivel, A., Sittig, D.F., Paul, L.A., Espadas, D., Singh, S., and Singh, H., Towards successful coordination of electronic health record based-referrals: A qualitative analysis. Implement Sci. 6(1):84, 2011. doi:10.1186/1748-5908-6-84.
Singh, H., Esquivel, A., Sittig, D.F., Murphy, D., Kadiyala, H., Schiesser, R., Espadas, D., and Petersen, L.A., Follow-up actions on electronic referral communication in a multispecialty outpatient setting. J Gen Intern Med. 26(1):64–69, 2011. doi:10.1007/s11606-010-1501-z.
Nielsen, J., Heuristic evaluation. In: Nielsen, J., and Mack, R.L. (Eds.), Usability inspection methods. Vol. 1. Wiley, New York, NY, pp. 25–62, 1994.
Thyvalikakath, T.P., Schleyer, T.K.L., and Monaco, V., Heuristic evaluation of clinical functions in four practice management systems: A pilot study. J Am Dent Assoc. 138(2):209–218, 2007. doi:10.14219/jada.archive.2007.0138.
Edwards, P.J., Moloney, K.P., Jacko, J.A., and Sainfort, F., Evaluating usability of a commercial electronic health record: A case study. Int J Hum Comput Stud. 66(10):718–728, 2008. doi:10.1016/j.ijhcs.2008.06.002.
Chan, J., Shojania, K.G., Easty, A.C., and Etchells, E.E., Usability evaluation of order sets in a computerised provider order entry system. BMJ Qual Saf. 20(11):932–940, 2011. doi:10.1136/bmjqs.2010.050021.
Nabovati, E., Vakili-Arki, H., Eslami, S., and Khajouei, R., Usability evaluation of laboratory and radiology information systems integrated into a hospital information system. J Med Syst. 38(4):35, 2014. doi:10.1007/s10916-014-0035-z.
Phansalkar, S., Zachariah, M., Seidling, H.M., Mendes, C., Volk, L., and Bates, D.W., Evaluation of medication alerts in electronic health records for compliance with human factors principles. J Am Med Inform Assoc. 21(e2):e332–e340, 2014. doi:10.1136/amiajnl-2013-002279.
Balatsoukas, P., Williams, R., Davies, C., Ainsworth, J., and Buchan, I., User Interface requirements for web-based integrated care pathways: Evidence from the evaluation of an online care pathway investigation tool. J Med Syst. 39(11):183, 2015. doi:10.1007/s10916-015-0357-5.
Lilholt, P.H., Jensen, M.H., and Hejlesen, O.K., Heuristic evaluation of a telehealth system from the Danish TeleCare north trial. Int J Med Inform. 84(5):319–326, 2015. doi:10.1016/j.ijmedinf.2015.01.012.
van Engen-Verheul, M.M., Peute, L.W.P., de Keizer, N.F., Peek, N., and Jaspers, M.W.M., Optimizing the user interface of a data entry module for an electronic patient record for cardiac rehabilitation: A mixed method usability approach. Int J Med Inform. 87:15–26, 2016. doi:10.1016/j.ijmedinf.2015.12.007.
Xu, L., Wen, D., Zhang, X., and Lei, J., Assessing and comparing the usability of Chinese EHRs used in two Peking University hospitals to EHRs used in the US: A method of RUA. Int J Med Inform. 89:32–42, 2016. doi:10.1016/j.ijmedinf.2016.02.008.
Wu, J., Militello, L. G., Flanagan, M. E., Barker, B. C., Rehman, S., Porter, B. W., Adams, J. M., Savoy, A. W., Russ, A. L., Weiner, M., Barriers and facilitators to using electronic health records for referrals between primary and specialty care clinics. Paper presented at the AMIA 2016 Annual Symposium, Chicago, IL, 2016
Shneiderman, B., Guidelines, principles, and theories In: Shneiderman B, Plaisant C, Cohen M, Jacobs S (eds) Designing the User Interface: Strategies for Effective Human-Computer Interaction, 5th edn. Addison-Wesley, Boston, MA, pp. 73–112, 2010.
Preece, J., Sharp, H., Rogers, Y., What is interaction design? In: Interaction Design: Beyond Human-Computer Interaction. 4th ed. edn. Wiley, Chichester, United Kingdom, pp 19-22, 2015.
Savoy, A., Yinni, G., and Salvendy, G., Effects of importance and detectability of usability problems on sample size requirements. Int J Hum Comput Interact. 25(5):430–440, 2009. doi:10.1080/10447310902865024.
Zhang, J., Johnson, T.R., Patel, V.L., Paige, D.L., and Kubose, T., Using usability heuristics to evaluate patient safety of medical devices. J Biomed Inform. 36(1–2):23–30, 2003. doi:10.1016/S1532-0464(03)00060-1.
Saldaña, J., An introduction to codes and coding. In: The coding manual for qualitative researchers. 3 edn. Sage, Los Angeles, CA, 2015.
Tarrell, A., Grabenbauer, L., McClay, J., Windle, J., and Fruhling, A.L., Toward improved heuristic evaluation of EHRs. Health Systems. 4(2):138–150, 2015. doi:10.1057/hs.2014.19.
Esquivel, A., Sittig, D.F., Murphy, D.R., and Singh, H., Improving the effectiveness of electronic health record-based referral processes. BMC Med Inform Decis Mak. 12(1):107, 2012. doi:10.1186/1472-6947-12-107.
Chen, A.H., Murphy, E.J., and Yee Jr., H.F., eReferral — A new model for integrated care. N Engl J Med. 368(26):2450–2453, 2013. doi:10.1056/NEJMp1215594.
Tuot, D.S., Leeds, K., Murphy, E.J., Sarkar, U., Lyles, C.R., Mekonnen, T., and Chen, A.H., Facilitators and barriers to implementing electronic referral and/or consultation systems: a qualitative study of 16 health organizations. BMC Health Serv Res. 15:568, 2015. doi:10.1186/s12913-015-1233-1.
Straus S.G., Chen A.H., Yee H.F., Jr., Kushel M.B., Bell D.S., Implementation of an electronic referral system for outpatient specialty care. In: AMIA 35th Annual Symposium on Biomedical and Health Informatics. American Medical Informatics Association, Washington, DC, pp 1337–1346, 2011
Warren, J., White, S., Day, K.J., Gu, Y., and Pollock, M., Introduction of electronic referral from community associated with more timely review by secondary services. Appl Clin Inform. 2(4):546–564, 2011. doi:10.4338/ACI-2011-06-RA-0039.
Chen, A.H., and Yee Jr., H.F., Improving the primary care–specialty care interface: Getting from here to there. Arch Intern Med. 169(11):1024–1026, 2009. doi:10.1001/archinternmed.2009.140.
Flanagan, M.E., Saleem, J.J., Millitello, L.G., Russ, A.L., and Doebbeling, B.N., Paper- and computer-based workarounds to electronic health record use at three benchmark institutions. J Am Med Inform Assoc. 20(e1):e59–e66, 2013. doi:10.1136/amiajnl-2012-000982.
Allen, M., Currie, L.M., Bakken, S., Patel, V.L., and Cimino, J.J., Heuristic evaluation of paper-based web pages: A simplified inspection usability methodology. J Biomed Inform. 39(4):412–423, 2006. doi:10.1016/j.jbi.2005.10.004.
Acknowledgments
We are thankful to the clinicians, research assistants, and clinical application coordinators at the three VA sites that assisted us in data collection. In addition, we are grateful to Jason J. Saleem, PhD for writing the grant proposal and generating study ideas and Laura G. Militello for piloting the methods.
Author information
Authors and Affiliations
Contributions
A.W. Savoy and H. Patel designed the heuristic evaluation, collected and analyzed data, and drafted the manuscript. M.E. Flanagan coauthored the funded grant proposal and collected and analyzed data. M. Weiner and A.L. Russ coauthored the funded grant proposal, and A.L. Russ proposed the manuscript on heuristic evaluation. All authors interpreted the findings, made critical revisions, and approved the published manuscript; all authors agree to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.
Corresponding author
Ethics declarations
Funding
This study was supported by VA HSR&D Grant IIR 12–102 (Principal Investigator: Dr. M. Weiner) and the Center for Health Information and Communication, US Department of Veterans Affairs, Veterans Health Administration, Health Services Research and Development Service, CIN 13–416, Richard L. Roudebush VA Medical Center, Indianapolis, Indiana. Dr. Russ was supported by a VA HSR&D Career Development Award, #11–214. M. Weiner is Chief of Health Services Research and Development at the Richard L. Roudebush VA Medical Center in Indianapolis, Indiana. The views expressed in this article are those of the authors and do not necessarily represent the views of the Department of Veterans Affairs.
Conflict of Interest
The authors have no competing interests to declare.
Ethical Approval
This article contains no studies with human participants or animals.
Additional information
This article is part of the Topical Collection on Systems-Level Quality Improvement
Electronic supplementary material
ESM 1
(DOCX 223 kb)
Rights and permissions
About this article
Cite this article
Savoy, A., Patel, H., Flanagan, M.E. et al. Systematic Heuristic Evaluation of Computerized Consultation Order Templates: Clinicians’ and Human Factors Engineers’ Perspectives. J Med Syst 41, 129 (2017). https://doi.org/10.1007/s10916-017-0775-7
Received:
Accepted:
Published:
DOI: https://doi.org/10.1007/s10916-017-0775-7