There is no “too small” for frequent workplace-based assessment: Differences between large and small residency programs in anesthesia when using a mobile application to assess EPAs; Es gibt kein „zu klein“ für häufige arbeitsplatzbasierte Assessments: Unterschiede zwischen großen und kleinen Weiterbildungsstätten in der Anästhesie beim Einsatz einer mobilen App zur Bewertung von EPAs

Abstract

Background: A competency-based education approach calls for frequent workplace-based assessments (WBA) of Entrustable Professional Activities (EPAs). While mobile applications increase the efficiency, it is not known how many assessments are required for reliable ratings and whether the concept can be implemented in all sizes of residency programs. Methods: Over 5 months, a mobile app was used to assess 10 different EPAs in daily clinical routine in Swiss anesthesia departments. The data from large residency programs was compared to those from smaller ones. We applied generalizability theory and decision studies to estimate the minimum number of assessments needed for reliable assessments. Results: From 28 residency programs, we included 3936 assessments by 306 supervisors for 295 residents. The median number of assessments per trainee was 8, with a median of 4 different EPAs assessed by 3 different supervisors. We found no statistically significant differences between large and small programs in the number of assessments per trainee, per supervisor, per EPA, the agreement between supervisors and trainees, and the number of feedback processes stimulated. The average “level of supervision” (LoS, scale from 1 to 5) recorded in larger programs was 3.2 (SD 0.5) compared to 2.7 (SD 0.4) (p<0.05). To achieve a g-coefficient >0.7, at least a random set of 3 different EPAs needed to be assessed, with each EPA rated at least 4 times by 4 different supervisors, resulting in a total of 12 assessments. Conclusion: Frequent WBAs of EPAs were feasible in large and small residency programs. We found no significant differences in the number of assessments performed. The minimum number of assessments required for a g-coefficient >0.7 was attainable in large and small residency programs. © 2024 Tessmann et al.

Keywords

Author Keywords

Competency-based education Feedback Decision Making reliability CBME entrustment EPA g theory mobile application residency program small business smartphone SME WBA

Index Keywords

clinical competence human medical education procedures Humans Internship and Residency workplace curriculum education Educational Measurement competency-based education Switzerland Education, Medical, Graduate Mobile applications anesthesiology mobile application

Author Affiliations

Institute of Anaesthesiology, UniversitatsSpital Zurich, Zurich, ZH, Switzerland

Intensive Care and Pain Medicine, Uniklinik Balgrist, Zurich, ZH, Switzerland

Institute for Medical Education, University of Bern, Bern, BE, Switzerland

Funding & Acknowledgements

No funding information

References 10 References

1 Frank, Jason R., Competency-based medical education: Theory to practice, Medical Teacher, 32, 8, pp. 638-645, (2010)

2 Assessing Competence in Professional Performance Across Disciplines and Professions, (2016)

3 Jonker, Gersten, A case for competency-based anaesthesiology training with entrustable professional activities: An agenda for development and research, European Journal of Anaesthesiology, 32, 2, pp. 71-76, (2015)

4 Breckwoldt, Jan, Entrustable professional activities: Promising concept in postgraduate medical education, Anaesthesist, 67, 6, pp. 452-457, (2018)

5 Englander, Robert M., From theory to practice: Making entrustable professional activities come to life in the context of milestones, Academic Medicine, 89, 10, pp. 1321-1323, (2014)

6 ten Cate, Olle Th J., Entrustability of professional activities and competency-based training, Medical Education, 39, 12, pp. 1176-1177, (2005)

7 Weller, Jennifer M., Can i leave the theatre? A key to more reliable workplace-based assessment, British Journal of Anaesthesia, 112, 6, pp. 1083-1091, (2014)

8 Pinilla, Severin, Entrustment versus performance scale in high-stakes OSCEs: Rater insights and psychometric properties, Medical Teacher, 45, 8, pp. 885-892, (2023)

9 Norcini, John J., 2018 Consensus framework for good assessment, Medical Teacher, 40, 11, pp. 1102-1109, (2018)

10 Schuwirth, Lambert W.T., Programmatic assessment: From assessment of learning to assessment for learning, Medical Teacher, 33, 6, pp. 478-485, (2011)

How to Cite