Skip to main content

Advertisement

Springer Nature Link
Log in

Systematic Heuristic Evaluation of Computerized Consultation Order Templates: Clinicians’ and Human Factors Engineers’ Perspectives

  • Systems-Level Quality Improvement
  • Published:
Journal of Medical Systems Aims and scope Submit manuscript

Abstract

We assessed the usability of consultation order templates and identified problems to prioritize in design efforts for improving referral communication. With a sample of 26 consultation order templates, three evaluators performed a usability heuristic evaluation. The evaluation used 14 domain-independent heuristics and the following three supplemental references: 1 new domain-specific heuristic, 6 usability goals, and coded clinicians’ statements regarding ease of use for 10 sampled templates. Evaluators found 201 violations, a mean of 7.7 violations per template. Minor violations outnumbered major violations almost twofold, 115 (57%) to 62 (31%). Approximately 68% of violations were linked to 5 heuristics: aesthetic and minimalist design (17%), error prevention (16%), consistency and standards (14%), recognition rather than recall (11%), and meet referrers’ information needs (10%). Severe violations were attributed mostly to meet referrers’ information needs and recognition rather than recall. Recorded violations yielded potential negative consequences for efficiency, effectiveness, safety, learnability, and utility. Evaluators and clinicians demonstrated 80% agreement in usability assessment. Based on frequency and severity of usability heuristic violations, the consultation order templates reviewed may impede clinical efficiency and risk patient safety. Results support the following design considerations: communicate consultants’ requirements, facilitate information seeking, and support communication. While the most frequent heuristic violations involved interaction design and presentation, the most severe violations lacked information desired by referring clinicians. Violations related to templates’ inability to support referring clinicians’ information needs had the greatest potential negative impact on efficiency and safety usability goals. Heuristics should be prioritized in future design efforts.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

References

  1. Kirsh, S.R., Ho, P.M., and Aron, D.C., Providing specialty consultant expertise to primary care: An expanding spectrum of modalities. Mayo Clin Proc. 89(10):1416(1411), 2014. doi:10.1016/j.mayocp.2014.04.016.

    Article  Google Scholar 

  2. Barnett, M.L., Song, Z., and Landon, B.E., Trends in physician referrals in the United States, 1999-2009. Arch Intern Med. 172(2):163–170, 2012. doi:10.1001/archinternmed.2011.722.

    Article  PubMed  PubMed Central  Google Scholar 

  3. Mehrotra, A., Forrest, C.B., and Lin, C.Y., Dropping the baton: Specialty referrals in the United States. Milbank Q. 89(1):39–68, 2011. doi:10.1111/j.1468-0009.2011.00619.x.

    Article  PubMed  PubMed Central  Google Scholar 

  4. Zuchowski, J.L., Rose, D.E., Hamilton, A.B., Stockdale, S.E., Meredith, L.S., Yano, E.M., Rubenstein, L.V., and Cordasco, K.M., Challenges in referral communication between VHA primary care and specialty care. J Gen Intern Med. 30(3):305–311, 2015. doi:10.1007/s11606-014-3100-x.

    Article  PubMed  Google Scholar 

  5. O’Malley, A.S., and Reschovsky, J.D., Referral and consultation communication between primary care and specialist physicians: Finding common ground. Arch Intern Med. 171(1):56–65, 2011. doi:10.1001/archinternmed.2010.480.

    PubMed  Google Scholar 

  6. Conley, J., Jordan, M., and Ghali, W.A., Audit of the consultation process on general internal medicine services. Quality & Safety in Health Care. 18(1):59–62, 2009. doi:10.1136/qshc.2007.025486.

    Article  CAS  Google Scholar 

  7. Salerno, S.M., Hurst, F.P., Halvorson, S., and Mercado, D.L., Principles of effective consultation: An update for the 21st-century consultant. Arch Intern Med. 167(3):271–275, 2007. doi:10.1001/archinte.167.3.271.

    Article  PubMed  Google Scholar 

  8. Saleem, J.J., Russ, A.L., Neddo, A., Blades, P.T., Doebbeling, B.N., and Foresman, B.H., Paper persistence, workarounds, and communication breakdowns in computerized consultation management. Int J Med Inform. 80(7):466–479, 2011. doi:10.1016/j.ijmedinf.2011.03.016.

    Article  PubMed  Google Scholar 

  9. Hysong, S.J., Esquivel, A., Sittig, D.F., Paul, L.A., Espadas, D., Singh, S., and Singh, H., Towards successful coordination of electronic health record based-referrals: A qualitative analysis. Implement Sci. 6(1):84, 2011. doi:10.1186/1748-5908-6-84.

    Article  PubMed  PubMed Central  Google Scholar 

  10. Singh, H., Esquivel, A., Sittig, D.F., Murphy, D., Kadiyala, H., Schiesser, R., Espadas, D., and Petersen, L.A., Follow-up actions on electronic referral communication in a multispecialty outpatient setting. J Gen Intern Med. 26(1):64–69, 2011. doi:10.1007/s11606-010-1501-z.

    Article  PubMed  Google Scholar 

  11. Nielsen, J., Heuristic evaluation. In: Nielsen, J., and Mack, R.L. (Eds.), Usability inspection methods. Vol. 1. Wiley, New York, NY, pp. 25–62, 1994.

    Google Scholar 

  12. Thyvalikakath, T.P., Schleyer, T.K.L., and Monaco, V., Heuristic evaluation of clinical functions in four practice management systems: A pilot study. J Am Dent Assoc. 138(2):209–218, 2007. doi:10.14219/jada.archive.2007.0138.

    Article  PubMed  Google Scholar 

  13. Edwards, P.J., Moloney, K.P., Jacko, J.A., and Sainfort, F., Evaluating usability of a commercial electronic health record: A case study. Int J Hum Comput Stud. 66(10):718–728, 2008. doi:10.1016/j.ijhcs.2008.06.002.

    Article  Google Scholar 

  14. Chan, J., Shojania, K.G., Easty, A.C., and Etchells, E.E., Usability evaluation of order sets in a computerised provider order entry system. BMJ Qual Saf. 20(11):932–940, 2011. doi:10.1136/bmjqs.2010.050021.

    Article  PubMed  Google Scholar 

  15. Nabovati, E., Vakili-Arki, H., Eslami, S., and Khajouei, R., Usability evaluation of laboratory and radiology information systems integrated into a hospital information system. J Med Syst. 38(4):35, 2014. doi:10.1007/s10916-014-0035-z.

    Article  PubMed  Google Scholar 

  16. Phansalkar, S., Zachariah, M., Seidling, H.M., Mendes, C., Volk, L., and Bates, D.W., Evaluation of medication alerts in electronic health records for compliance with human factors principles. J Am Med Inform Assoc. 21(e2):e332–e340, 2014. doi:10.1136/amiajnl-2013-002279.

    Article  PubMed  PubMed Central  Google Scholar 

  17. Balatsoukas, P., Williams, R., Davies, C., Ainsworth, J., and Buchan, I., User Interface requirements for web-based integrated care pathways: Evidence from the evaluation of an online care pathway investigation tool. J Med Syst. 39(11):183, 2015. doi:10.1007/s10916-015-0357-5.

    Article  PubMed  Google Scholar 

  18. Lilholt, P.H., Jensen, M.H., and Hejlesen, O.K., Heuristic evaluation of a telehealth system from the Danish TeleCare north trial. Int J Med Inform. 84(5):319–326, 2015. doi:10.1016/j.ijmedinf.2015.01.012.

    Article  PubMed  Google Scholar 

  19. van Engen-Verheul, M.M., Peute, L.W.P., de Keizer, N.F., Peek, N., and Jaspers, M.W.M., Optimizing the user interface of a data entry module for an electronic patient record for cardiac rehabilitation: A mixed method usability approach. Int J Med Inform. 87:15–26, 2016. doi:10.1016/j.ijmedinf.2015.12.007.

    Article  PubMed  Google Scholar 

  20. Xu, L., Wen, D., Zhang, X., and Lei, J., Assessing and comparing the usability of Chinese EHRs used in two Peking University hospitals to EHRs used in the US: A method of RUA. Int J Med Inform. 89:32–42, 2016. doi:10.1016/j.ijmedinf.2016.02.008.

    Article  PubMed  Google Scholar 

  21. Wu, J., Militello, L. G., Flanagan, M. E., Barker, B. C., Rehman, S., Porter, B. W., Adams, J. M., Savoy, A. W., Russ, A. L., Weiner, M., Barriers and facilitators to using electronic health records for referrals between primary and specialty care clinics. Paper presented at the AMIA 2016 Annual Symposium, Chicago, IL, 2016

  22. Shneiderman, B., Guidelines, principles, and theories In: Shneiderman B, Plaisant C, Cohen M, Jacobs S (eds) Designing the User Interface: Strategies for Effective Human-Computer Interaction, 5th edn. Addison-Wesley, Boston, MA, pp. 73–112, 2010.

    Google Scholar 

  23. Preece, J., Sharp, H., Rogers, Y., What is interaction design? In: Interaction Design: Beyond Human-Computer Interaction. 4th ed. edn. Wiley, Chichester, United Kingdom, pp 19-22, 2015.

  24. Savoy, A., Yinni, G., and Salvendy, G., Effects of importance and detectability of usability problems on sample size requirements. Int J Hum Comput Interact. 25(5):430–440, 2009. doi:10.1080/10447310902865024.

    Article  Google Scholar 

  25. Zhang, J., Johnson, T.R., Patel, V.L., Paige, D.L., and Kubose, T., Using usability heuristics to evaluate patient safety of medical devices. J Biomed Inform. 36(1–2):23–30, 2003. doi:10.1016/S1532-0464(03)00060-1.

    Article  PubMed  Google Scholar 

  26. Saldaña, J., An introduction to codes and coding. In: The coding manual for qualitative researchers. 3 edn. Sage, Los Angeles, CA, 2015.

  27. Tarrell, A., Grabenbauer, L., McClay, J., Windle, J., and Fruhling, A.L., Toward improved heuristic evaluation of EHRs. Health Systems. 4(2):138–150, 2015. doi:10.1057/hs.2014.19.

    Article  Google Scholar 

  28. Esquivel, A., Sittig, D.F., Murphy, D.R., and Singh, H., Improving the effectiveness of electronic health record-based referral processes. BMC Med Inform Decis Mak. 12(1):107, 2012. doi:10.1186/1472-6947-12-107.

    Article  PubMed  PubMed Central  Google Scholar 

  29. Chen, A.H., Murphy, E.J., and Yee Jr., H.F., eReferral — A new model for integrated care. N Engl J Med. 368(26):2450–2453, 2013. doi:10.1056/NEJMp1215594.

    Article  CAS  PubMed  Google Scholar 

  30. Tuot, D.S., Leeds, K., Murphy, E.J., Sarkar, U., Lyles, C.R., Mekonnen, T., and Chen, A.H., Facilitators and barriers to implementing electronic referral and/or consultation systems: a qualitative study of 16 health organizations. BMC Health Serv Res. 15:568, 2015. doi:10.1186/s12913-015-1233-1.

    Article  PubMed  PubMed Central  Google Scholar 

  31. Straus S.G., Chen A.H., Yee H.F., Jr., Kushel M.B., Bell D.S., Implementation of an electronic referral system for outpatient specialty care. In: AMIA 35th Annual Symposium on Biomedical and Health Informatics. American Medical Informatics Association, Washington, DC, pp 1337–1346, 2011

  32. Warren, J., White, S., Day, K.J., Gu, Y., and Pollock, M., Introduction of electronic referral from community associated with more timely review by secondary services. Appl Clin Inform. 2(4):546–564, 2011. doi:10.4338/ACI-2011-06-RA-0039.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  33. Chen, A.H., and Yee Jr., H.F., Improving the primary care–specialty care interface: Getting from here to there. Arch Intern Med. 169(11):1024–1026, 2009. doi:10.1001/archinternmed.2009.140.

    Article  PubMed  Google Scholar 

  34. Flanagan, M.E., Saleem, J.J., Millitello, L.G., Russ, A.L., and Doebbeling, B.N., Paper- and computer-based workarounds to electronic health record use at three benchmark institutions. J Am Med Inform Assoc. 20(e1):e59–e66, 2013. doi:10.1136/amiajnl-2012-000982.

    Article  PubMed  PubMed Central  Google Scholar 

  35. Allen, M., Currie, L.M., Bakken, S., Patel, V.L., and Cimino, J.J., Heuristic evaluation of paper-based web pages: A simplified inspection usability methodology. J Biomed Inform. 39(4):412–423, 2006. doi:10.1016/j.jbi.2005.10.004.

    Article  PubMed  Google Scholar 

Download references

Acknowledgments

We are thankful to the clinicians, research assistants, and clinical application coordinators at the three VA sites that assisted us in data collection. In addition, we are grateful to Jason J. Saleem, PhD for writing the grant proposal and generating study ideas and Laura G. Militello for piloting the methods.

Author information

Authors and Affiliations

  1. Center for Health Information and Communication, US Department of Veterans Affairs, Veterans Health Administration, Health Services Research and Development Service CIN 13-416, Richard L. Roudebush VA Medical Center, Indianapolis, IN, USA

    April Savoy, Himalaya Patel, Michael Weiner & Alissa L. Russ

  2. Regenstrief Institute, Inc., Indianapolis, IN, USA

    April Savoy, Michael Weiner & Alissa L. Russ

  3. Indiana University East School of Business and Economics, Richmond, IN, USA

    April Savoy

  4. Indiana University School of Medicine, Indianapolis, IN, USA

    Mindy E. Flanagan & Michael Weiner

  5. Department of Pharmacy Practice, College of Pharmacy, Purdue University, West Lafayette, IN, USA

    Alissa L. Russ

Authors
  1. April Savoy
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Himalaya Patel
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Mindy E. Flanagan
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Michael Weiner
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Alissa L. Russ
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

A.W. Savoy and H. Patel designed the heuristic evaluation, collected and analyzed data, and drafted the manuscript. M.E. Flanagan coauthored the funded grant proposal and collected and analyzed data. M. Weiner and A.L. Russ coauthored the funded grant proposal, and A.L. Russ proposed the manuscript on heuristic evaluation. All authors interpreted the findings, made critical revisions, and approved the published manuscript; all authors agree to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.

Corresponding author

Correspondence to April Savoy.

Ethics declarations

Funding

This study was supported by VA HSR&D Grant IIR 12–102 (Principal Investigator: Dr. M. Weiner) and the Center for Health Information and Communication, US Department of Veterans Affairs, Veterans Health Administration, Health Services Research and Development Service, CIN 13–416, Richard L. Roudebush VA Medical Center, Indianapolis, Indiana. Dr. Russ was supported by a VA HSR&D Career Development Award, #11–214. M. Weiner is Chief of Health Services Research and Development at the Richard L. Roudebush VA Medical Center in Indianapolis, Indiana. The views expressed in this article are those of the authors and do not necessarily represent the views of the Department of Veterans Affairs.

Conflict of Interest

The authors have no competing interests to declare.

Ethical Approval

This article contains no studies with human participants or animals.

Additional information

This article is part of the Topical Collection on Systems-Level Quality Improvement

Electronic supplementary material

ESM 1

(DOCX 223 kb)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Savoy, A., Patel, H., Flanagan, M.E. et al. Systematic Heuristic Evaluation of Computerized Consultation Order Templates: Clinicians’ and Human Factors Engineers’ Perspectives. J Med Syst 41, 129 (2017). https://doi.org/10.1007/s10916-017-0775-7

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s10916-017-0775-7

Keywords

Search

Navigation