Competence by Design Innovators

Given the size of the Psychiatry program, we were well-positioned to try some novel approaches, learn from them, and ultimately inform the national approach to our specialty. The Royal College approved our Fundamental Innovations in Residency Education (FIRE) application. In 2016 and 2017 we ran a CBME PGY1 pilot and learned a great deal about the practicalities of an assessment program that includes workplace-based assessments. In 2018, we had a PGY2 pilot group, and each year we expanded the pilot to include another PG year until we had some experience across all five training years.

PsychRocks is a home-grown electronic platform we created when starting our pilot, to allow for the collection of multiple low-stakes assessments such as EPAs. We knew that it would be a nightmare to implement CBD using pen and paper. The availability of thousands of pieces of data, as well as the completion, collation, storage, and review by the Competence Committee, needed to be done electronically. We stipulated that PsychRocks needed to be widely available and would reside primarily with the residents. The plan was for residents to initiate assessments and then hand their phones to their workplace supervisor so they could have timely assessments completed in the clinical setting. This was pre-COVID when social distancing was not yet a consideration. While faculty could still initiate assessments, this setup allowed residents to have more autonomy and responsibility for their own education and assessment.

We employed student developers, so the cost to build the platform was very low. After brainstorming sessions to determine needs and the flow of information, we had a group of faculty and student volunteers walk through the process. The programmers designed the platform to meet the identified needs. For example, the programmers were able to develop a system that allowed select assessors including nurses, administrators, allied health, etc. to provide assessments, and ensure that assessments were completed accurately. Ultimately, we used PsychRocks for several years until the university was able to provide a suitable platform for all programs that coincided with the official Royal College roll-out of CBME. PsychRocks was incredibly useful in helping us understand how EPA assessments function on the ground. It also helped students and faculty learn about the practicalities of the tool and the educational principles behind CBME assessments.

PsychRocks was always intended to be temporary until the Royal College or the university could supply an appropriate electronic platform. Through its development, we learned so much about CBME including how to manage and present data, and how to encourage timely completion of EPAs in the clinical setting. It was an excellent data collection tool. However, it had a limited “dashboard” functionality. It became clear over time that residents, faculty, and competence committees would require more information about how many assessments had been completed, how many requirements were left, trajectory over time, etc. Aspects of platforms currently being used across the country are still being refined. So, while PsychRocks was essential to help us start our CBD journey, the resources required to refine, improve and troubleshoot difficulties over time would be more than any single program could manage. It was beneficial to transition to a university-wide platform when it became available, which allowed us to focus on other aspects of CBME rollout.

We learned that no matter how much you are communicating – it is never enough! We also learned about the importance of having multiple modalities and opportunities for faculty and resident development. For example, we used regular email updates, branded monthly newsletters, in-person sessions, online sessions, videos for asynchronous learning and monthly contests for the most EPAs completed. We also had a myths and facts session which was helpful in addressing any rumours. As the rollout proceeded, we began a weekly email that addressed questions submitted by residents and helped to demonstrate the responsive nature of the program.

We are in a specialty that has a good track record of providing in the moment feedback, even prior to CBD. Initially, some residents and faculty complained that those formative conversations were already happening and the paperwork was perceived as a burden. However, as the Competence Committee began to review files, they were able to spot trends that hadn’t necessarily been apparent to individual supervisors or on a given rotation. Many of the other benefits were more adjacent to CBD. For example, developing EPAs locally and then nationally required a review of the specialty to better define what we want psychiatric practice and practitioners to look like in the future. We also embraced the use of technology including electronic portfolios, and Zoom for learning, which helped modernize our program. These changes also allowed us to adapt more easily to the pandemic as we already had experience with technology that helped to enable social distancing without sacrificing the quality of education.