In July 2022, the AAMC convened a two-day workshop to create evidence-based action plans for four complex and urgent challenges during the transition to residency: away rotations, the GME interactive informational database, holistic review in residency selection, and interview protocols. This report summarizes the proceedings, discussions, priority activities, and next steps from the workshop.
On This Page:
Executive Summary
In July 2022, the AAMC convened a two-day workshop to create evidence-based action plans for four complex and urgent challenges during the transition to residency: away rotations (UGRC rec. 13*); the GME interactive informational database (UGRC rec. 6); holistic review in residency selection (UGRC recs. 4-5); and interview protocols. The four areas identified are those that are most timely and in need of broader input and further action planning by the wider community. The attendees represented national organizations, specialty societies, and AAMC constituent groups across the undergraduate medical education (UME)-graduate medical education (GME) continuum. This report summarizes the proceedings, discussions, priority activities, and next steps from the workshop.
*Undergraduate Medical Education-Graduate Medical Education Review Committee (UGRC) report
Day One: Summary
The workshop used a modified World Café format and began with a welcome from AAMC Chief Academic Officer Alison J. Whelan, MD, and AAMC Chief Services Officer Gabrielle V. Campbell, MBA, LLM. During their remarks, they outlined the ground rules for the workshop. Dr. Whelan encouraged attendees to “leave their official titles at the door” and to participate not as representatives of their respective organizations but as themselves and the part of the transition to residency community they reflect (e.g., students, UME, GME, medical schools, programs). She asked them to keep an open mind as they shared and received perspectives from fellow participants and expressed hope that everyone would be open to the event format and share their views and ideas candidly in what was intended to be a safe space.
After group introductions, Dr. Whelan introduced the first part of the day’s activities. The goal for participants, as subject matter experts, was to consider the following questions for each of the identified topic areas:
-
What is currently being done to address the challenges in this area?
-
What groups are involved and what is underway?
-
What is known?
-
What questions remain?
Dr. Whelan emphasized the need to identify the gaps that future action plans could address and to refrain from skipping straight to solutions (which would be the focus of Day Two).
Finally, she asked the attendees to split into four groups (which were not predetermined or self-selected), and she randomly assigned a topic to each group before beginning the Gallery Walk and first interactive exercises.
The Gallery Walk aimed to give all participants a common understanding of the current state of the four topic areas and an understanding of the intended scope of work for the brainstorm and action planning activities to follow. The information about the four areas was set up on easels at four stations around the room. Each station included data and statistics highlighting the current landscape, as well as what is presently underway to address the area’s challenges based on activities submitted by the participants prior to the meeting. AAMC group facilitators answered questions as attendees reviewed and reacted to the data.
Once everyone had time to consider the data at all four Gallery Walk stations, the facilitators began the next section of the workshop: the four interactive table topics. They used flip charts to capture the groups’ feedback and ideas on what is underway and what is still missing. The purpose for this stage of the workshop was to explore the potential actions rather than dive into details, the feasibility of the actions, or how to assign future tasks to specific organizations. In most cases, groups had time to indicate what they felt were the most important issues that needed to be addressed in the future. Some groups also had time to discuss short-, medium-, and long-term priorities.
Participants rotated through two of the four interactive table topics before facilitators provided an interim update on the progress made midway through the exercise. Dr. Whelan asked hosts and attendees to share what surprised them, what was still missing, and what common themes came up in the four discussions.
The AAMC hosts ended the day by thanking everyone for being part of the dialogue and urging all to arrive for Day Two ready to complete the interactive exercise for the two remaining topics they hadn’t yet completed — and to begin thinking through solutions.
Day Two: Summary
After a brief recap of the process and updates from Day One, the four groups rotated through the interactive exercises for the two remaining topics they had not completed the day before.
Throughout the interactive exercise rounds, the facilitators generated a working list of actions for further consideration. They then condensed and combined attendee ideas onto flip chart listings so the groups could determine which ideas were the priority. The groups indicated on the charts which actions they felt were most important.
Throughout the day, attendees had an opportunity to leave their original groups and join another topic group of interest. The groups reviewed all the prioritized actions and discussed items that needed clarification and elaboration. Dr. Whelan and Ms. Campbell reminded the attendees that they are not committing themselves or any particular organization to tasks or actions beyond the workshop (assignments or next steps were not part of this exercise).
Near the end of Day Two, workshop attendees were asked to review all action plan activities and further refine what they viewed as the priority next steps and then vote for their top three items in each of the four topic areas. AAMC staff did not vote.
The AAMC hosts concluded the event with a commitment to continue the discussions that were introduced by workshop participants. They thanked the attendees for their engaged and expert participation but also assured them that they would not be tasked with participating in any follow-on activities or attending additional events (unless they chose to).
Group Discussions
This section of the report summarizes the discussion points from both days of the workshop. Day One focused on identifying the current state of the transition to residency, what is already underway to address challenges, and what questions remain. Day Two activities turned to what solutions/actions could begin to address challenges and improve the transition to residency.
Away Rotations
The away rotation discussions all began with the question, “How might we catalyze the work to improve the quality of and access to away rotations?” Attendees discussed the inconsistent purpose of away rotations: Some saw them purely as part of the student's education while others saw them as auditions for residency. The points of view often differed between UME and GME representatives. They agreed that away rotations were essential for students who didn’t have access to specialty training at their own medical schools but wondered why the specialty requirements, as well as the length and cost of rotations, were so varied. Many agreed that it needs to be clear to students about what they will get out of an away rotation and clearer to programs/clinical sites about what they need to offer students.
By the end of the interactive exercises, it became clear that all parties need more data but don’t know what is already available to them. Attendees prioritized taking an inventory of existing data, gathering students' input on the cost burden, defining the purpose/priorities of away rotations to determine what future environment is needed, and encouraging specialties to clearly say what they need from students and why.
Based on the attendees’ final vote tally, the following action steps were identified as the highest priority:
-
Identify sources of existing data. There is no single source of comprehensive data about away rotations, and a student’s motivation to seek one or more away rotations can vary greatly. Therefore, the AAMC and others from the transition to residency community need to find data to support future away rotation discussions and recommended actions from sources across medical education. For example:
-
Student data: What is the percent of students who apply to away rotations? What is the percent of students who complete away rotations?
-
Medical school data: Was specialty training available at the school when a student was seeking a rotation? Is an away rotation a graduation requirement? Does the institution place limits on the number of away rotations to which students can apply and attend?
-
Clinical site data: Are there set criteria for reviewing and accepting students for away rotations? Is there a set student evaluation process?
-
Specialty data: Does the number of available specialty away rotation opportunities support the number of students seeking those rotations? Does the specialty organization provide clear guidance regarding the recommended/required number(s) of away rotations?
-
-
Determine the purpose(s) of away rotations. There are distinct and related reasons for a student to seek an away rotation opportunity. Based on data, including residency and professional data, consider how to create a future environment that clarifies and standardizes the intent and the outcome of away rotations.
-
Develop recommendations for addressing the cost burden of away rotations. Although we need to consider equity throughout the medical school learner journey, the cost burden of away rotations could also impact a student’s considerations. Geographic location (factoring in travel, transportation, housing, and meals), number of required specialty rotations, or if a student can even consider an away experience at all are all part of a student’s away rotation financial calculation.
-
Encourage development and dissemination of clear explanations of requirements and deliverables. Students may have less anxiety and make better-informed decisions if specialty organizations clearly present student requirements/recommendations for rotations. How many are expected and for what purpose? Host programs could also better support students if they clearly presented their requirements, decision process, and timeline and defined their program deliverables (e.g., letters of recommendation, mentors, interviews).
GME Interactive Informational Database
The GME interactive informative database discussions asked, “How might we ensure existing and developing GME program informational technologies are accessible, accurate, and user-friendly?
Workshop participants acknowledged the myriad tools and resources currently available for students to explore different programs and specialties. These tools and resources are provided by multiple organizations; differ in design, display, and functionality; and vary significantly in the way they source and maintain data and information. Data that is not maintained may be outdated or inaccurate, thereby making it less valuable or even useless to students. A lack of coordination or collaboration among organizations and data sources (with notable exceptions) contributes to the situation. Further, there is significant inconsistency across programs in what information they share on their websites or elsewhere, likely due to varying resources, motivation, and residency selection processes. As a result, students turn to other sources of information such as social media for feedback from other students; however, this information also varies in accuracy and quality.
Given these challenges, students, advisors, and programs noted the value in creating a single database to house information for all programs — one that includes a streamlined set of data that can inform applicants’ choice of where to apply. Importantly, it should help students identify programs where they could thrive (e.g., be satisfied, be engaged, align with programs’ values) and where they are likely to be competitive (specifically, where they are likely to receive an interview). To be valuable and promote transparency, the data must be accurate, reliable, complete, and up to date, which will require ongoing maintenance and accountability. To the extent possible, data and information should be drawn from existing data sources (e.g., systems, surveys) to avoid duplicating efforts. Steps must also be taken to encourage program participation and gain buy-in to providing information. There should be a streamlined, standardized collection of data and information from programs with the value of participation highlighted (e.g., opportunity to market and/or benchmark to other programs).
Based on the attendees’ final vote tally, the following action steps were identified as the highest priority:
- Create an inventory of all existing databases, tools, and resources. This inventory could be valuable for students to know where to look for certain data and information as the transition to residency community works toward a single GME database. It also allows the community to take stock of what data is currently available and from which data sources. From here, stakeholders can review what is currently available to determine its value and what may be missing.
- Study and define what it means to succeed or thrive in residency. To ensure databases are providing the most valuable data, we first must understand what it means for a student to thrive in residency and what factors contribute to that.
- Rethink program missions to ensure authenticity. A program’s mission is important information to applicants, but to be truly valuable, its mission needs to be authentic and truly reflect core tenets such as its values, goals, opportunities, and resources. Programs could benefit from toolkits and training as they rethink their missions.
- Engage many different stakeholders (e.g., students, advisors, programs) with representation from specific groups (e.g., groups underrepresented in medicine including gender, race/ethnicity, socioeconomic status, first-generation). Ensuring that many different perspectives are included in tools such as focus groups and surveys will help ensure the most valuable data is identified. Students vary (e.g., their characteristics, their needs), so the data and information they seek may also vary.
- Consider different data sources and stakeholders. Many different organizations are a part of the transition to residency space, each of which maintains different types of data that could be leveraged to create a valuable resource. Some of these organizations are already contributing data to existing databases. Further collaboration and coordination will be critical to this effort.
- Create a single database of all programs that includes standardized information. This database should contain a common set of reliable valid data for all programs. It should consider, as appropriate, data to describe the characteristics of applicants, interviewees, and residents of the program, as well as faculty and staff in the institution and patient population. It may also provide an option to link back to program websites or social media accounts.
Holistic Review
The holistic review in GME discussions asked, “How might we shift away from a reliance on metrics (license exam scores) to a broader, more holistic view of residents’ experiences and attributes?” Attendees spoke about strong theoretical support for holistic review, as well as about the many cultural and practical challenges that need to be addressed to increase implementation. Discussions about advising, the need for transparent and up-to-date GME databases, and more standardized interview timelines and processes also highlighted how interdependent holistic review is on the three other workshop focus areas.
Lack of trust and transparency were identified as persistent challenges by both UME and GME attendees. Numerous attendees raised concerns about the information provided by medical schools (e.g., academic readiness, performance data) and both the content and usability/functionality of the Electronic Residency Application Service® (ERAS®). Every group provided recommendations for ERAS program changes to facilitate a holistic review, including: making all data in ERAS searchable; aligning ERAS with experiences-attributes-academic model metrics; linking relevant American Medical College Application Service® indicators with ERAS (e.g., socioeconomic status, first-generation); providing decision support tools; allowing PDWS users to add weights to different categories; and reframing how we ask applicants to provide information to enable them to tell a story. Attendees also recommended running specialty-specific pilots in ERAS and disseminating outcomes with the broader UME-GME community, as well as ensuring that applicant groups are engaged before changes are made.
Attendees felt strongly that accrediting bodies need to be involved in these discussions and action planning. This belief connects with what attendees identified as an overall lack of true leadership accountability at the national level, which leads to recommendations being made without any means of enforcement. Additionally, they felt accreditation standards could be strengthened — or critically examined — and revised to facilitate adoption of holistic review.
The clear need for a broader and deeper evidence base for the impact and benefits of holistic review was also a persistent theme. GME attendees in particular emphasized this need, given both the time- and resource-intensiveness and patient safety priorities. This need was identified as work for national organizations, since much of the current evidence is from single school/program studies, as well as for specialty program director organizations and program leaders.
Attendees stressed the need to educate and build buy-in among leaders and faculty who might be peripheral to selection but play key roles in allocating resources, including protected time for those involved in screening and selection and who may also be serving as interviewers who create program/institutional climate and culture (e.g., department chairs, faculty, deans). Relatedly, attendees spoke about the importance of addressing program culture, environment, and learning and psychosocial supports, particularly if programs are bringing in new/different cohorts of learners.
Based on the attendees’ final vote tally, the following action steps were identified as the highest priority:
-
Build out ERAS to facilitate holistic review. Multiple suggestions are nested under this priority. On the program side, suggestions include providing data analytics and decision support tools for programs, making all ERAS data searchable, connecting data to the experiences-attributes-academic metrics model, adding SES and first-generation college indicators, and asking applicants to identify a small number of defining experiences. On the applicant side, suggestions include simplifying the inputs and reframing how information is requested (e.g., “Identify your top five most meaningful experiences,” “Identify themes”) to enable them to more effectively tell their story.
-
Create tools and resources for program staff. This would include practical tools (e.g., rubrics, training modules) and effective practices for integrating holistic principles into recruitment and selection processes by specialty/program type, as well as resources on change management, evaluation, and implicit and structural bias.
-
Identify levers to encourage and increase accountability. The main items in this category are to conduct research projects to better understand and document outcomes and to work with the Accreditation Council for Graduate Medical Education, including in the development and co-branding of resources and critical analysis of accreditation standards.
Interview Protocols
The interview considerations and protocols interactive table discussions addressed the question, “How might we develop and agree upon consistent logistical processes for interviewing applicants for residency programs?” At the beginning of Day One, given the essential role that interviews play in the residency application process, attendees first outlined issues around applications — not interviews. For example, there were hearty discussions about how overapplication (which workshop participants linked to advising, lack of transparency, and applicant fear) makes it hard for programs to enact holistic review; how UME and GME communities place too much emphasis on “how far people went down their rank list”; how differing (and vague) specialty guidelines aren’t consistently enforced and seem to confuse all parties more than help them; and how application caps would burden all but the most competitive students.
With the application issues aired, the attendees then discussed many issues related to interviews: The data needed to evaluate virtual, hybrid, and in-person interviewing; (perceived) interview hoarding; how to access and share historical, trusted interview data to help learners manage their interview offers; and how to ensure students can focus on learning — and not just recruitment — during the final year of medical school. Numerous conversations highlighted the power imbalance between specialties, programs, and institutions — and how that dynamic creates distrust, lack of transparency, and inequity.
During the second day of the workshop, attendees refocused on the data, unwritten rules, and resources that students, advisors, and programs need to make interviews an effective part of holistic review. Based on the attendees’ final vote tally, the following action steps were identified as the highest priority:
- Develop “traffic rules” to guide interview offers. New, clear guidance for all parties involved would provide transparent timelines; enable programs to offer the same number of interview invitations as available slots; standardize the release dates and response times; clarify pre- and post-interview communication rules; and have everyone adhere to the National Resident Matching Program® guidelines.
- Complete interview caps research. The research would help determine if interview caps impact the number of applications that learners submit or address interview hoarding (perceived or real).
- Encourage change in advisor behavior. In addition to educating faculty advisors about how to advise their students about overapplications, the community could also consider the impact of the Liaison Committee on Medical Education on how learners are advised and eliminate the tying of board passage with program accreditation.
- Explore long-term impact of virtual interviewing. Workshop attendees prioritized continuing to review data and assess/evaluate the impact of virtual interviewing on outcomes (e.g., match rate, diversity/equity). Future research also needs to evaluate how virtual interviewing compares to in-person or hybrid interviews and if in-person visits benefit students. Finally, the community can further discuss how to develop ways to assess professionalism virtually if in-person group and one-on-one interactions aren’t allowed. For example, do we need a situational judgement test for residency?
Note: Signaling did not get many votes. When attendees asked about the low vote total, the response from facilitators was that signaling was underway and the community is awaiting data to see what the impact would be. Participants were not inclined to waste a final vote for something favorable and already in progress.
Next Steps
It was clear from formal and informal discussion throughout the two days that common priority activities and conclusions emerged for all four topic areas:
- This is just the beginning of the process. The UGRC report was a significant step, but the entire community needs to work together to better understand what supports are currently available from various elements of the medical education community before embarking on new paths for improving the transition to residency. For example, there is a good deal of data already available (or being gathered as part of current research projects) that is not currently catalogued and can begin to inform communitywide decision making if shared more widely. A more comprehensive understanding of what data exists can also help refine and support the community priorities identified in the two-day workshop.
- Increased transparency, trust, and information sharing. To achieve timely and comprehensive improvements to the transitions to residency, there is an obvious need to state clearly what every stakeholder needs to do to enable learners to become thriving physicians. The community must all commit to being open with our expectations, using straightforward language, and collating our needs into easy-to-find and easy-to-use information sources that everyone trusts. This work will be challenging but achievable, and it was a clear priority for those workshop participants who are engaged in the various transition to residency efforts.
- The UME-GME community needs more data. As we inform and educate the community about existing data, we need to consider carefully what additional data is needed to inform any significant community changes or next steps. Is it a matter of compiling/integrating existing data? Or do we also begin new collaborative data collection activities to capture new types of useful data needed to support decision making aimed at improvements to the transition to residency?
- We must act together. Another common theme was the need for a collective response to the challenges we face. No one group alone can manage the complex actions needed to ensure successful, comprehensive actions that limit unintended consequences while supporting learners and the academic and professional institutions and organizations dedicated to creating more focused training for tomorrow’s physicians and researchers.
- Continue the dialogue. This workshop is an excellent example of the power and direct usefulness of continued dialogue among the medical education community as we strive to improve the complex transition to residency process in ways beneficial to students and the academic community that supports their learning and training. We need to continue, in forums large and small, to intentionally engage with one another to ensure successful improvements are achieved and established avenues for continued dialogue are in place.
Meeting Attendees
Jerel Arceneaux, MS
Sr. Financial Aid Officer
University of South Carolina
Jesse Burk-Rafel, MD
Assistant Director of UME-GME Innovation, Institute for Innovations in Medical Education
NYU Grossman School of Medicine
Rachel Eleazu
National Vice President, Student National Medical Association (SNMA)
MD candidate, University of Pittsburgh School of Medicine
Mary Furlong, MD
Associate Dean for Curriculum and Director of the Office of Medical Education
Georgetown University School of Medicine
Chris Gallo, MD
Representative for the Association of Native American Medical Students (ANAMS)
Resident, Duke University
Maya M. Hammoud, MD, MBA
Representative for the Association of Professors of Gynecology & Obstetrics (APGO)
Professor and Associate Chair for E-Learning and Enabling Technologies, Obstetrics and Gynecology
University of Michigan Medical School
Leila E. Harrison, PhD, MA, MEd
Senior Associate Dean for Admissions and Student Affairs
Washington State University Elson S. Floyd College of Medicine
Dilpreet Kaeley
Chair-Elect, AAMC Organization of Student Representatives
MD candidate, University of Toledo College of Medicine and Life Sciences
Kathleen J. Kashima, PhD
Senior Associate Dean of Students
University of Illinois Chicago
Donna L. Lamb, DHSc, MBA, BSN
President and CEO
National Resident Matching Program
Kimberly Lomis, MD
Vice President for UME Innovations
American Medical Association
LaTanya J. Love, MD
Dean of Education, ad interim
Associate Professor, Pediatrics
UT Health Science Center at Houston
Elise Lovell, MD
Immediate Past President, Organization of Program Director Associations (OPDA)
Program Director, Emergency Medicine and Clinical Professor
Advocate Christ Medical Center, University of Illinois at Chicago
Hilit F. Mechaber, MD
Senior Associate Dean for Student Affairs
University of Miami Miller School of Medicine
Sunny Nakae, MSW, PhD
Senior Associate Dean – Equity, Inclusion, Diversity, and Partnership
California University of Science and Medicine
William Pieratt, DO, FACP
Representative for American Association of Colleges of Osteopathic Medicine (AACOM)
Dean and Chief Academic Officer
Burrell College of Osteopathic Medicine
Jeanmarie Rey, MD
Representative for the American Academy of Family Physicians (AAFP)
Assistant Professor, Family Medicine and Director of Reflective Practice
Uniformed Services University
Sandra Snyder, DO
Representative for American Association of Colleges of Osteopathic Medicine (AACOM)
Program Director, Family Medicine
Cleveland Clinic
Deborah Spitz, MD
Representative for the Organization of Program Director Associations (OPDA)
Vice Chair for Education and Academic Affairs
Professor of Psychiatry and Behavioral Neuroscience
The University of Chicago
Kyla P. Terhune, MD, MBA, FACS
Representative for Assoc of Program Directors in Surgery (APDS)
Associate Dean for Graduate Medical Education
Vanderbilt University School of Medicine
Simone Thavaseelan, MD
Representative for the Society of Academic Urology (SAU)
Associate Professor of Surgery (Urology)
Warren Alpert School of Medicine at Brown University
Lia A. Thomas, MD
Representative of the American Association of Directors of Psychiatric Residency Training (AADPRT)
Medical Director, MH Trauma Services Team, VA North Texas Health Care System
UT Southwestern Medical Center
Donna Tran
President, Asian Pacific American Medical Student Assoc (APAMSA)
MD student at Michigan State University College of Human Medicine and
MPH student at Johns Hopkins School of Public Health
Teresa A. Vigil, MD
Interim Senior Associate Dean of Education
University of New Mexico
Linda A. Waggoner-Fountain, MD, MEd
Representative for the Association of Pediatric Program Directors (APPD)
Director, Pediatrics Residency Program
UVA Health
David A. Wininger, MD
Representative for the Alliance for Academic Internal Medicine (AAIM)
IM Residency Program Director
Ohio State University Wexner Medical Center
Fasika Woreta, MD
Representative for the Association of University Professors of Ophthalmology (AUPO)
Director, Ophthalmology Residency Program
Associate Professor of Ophthalmology
Johns Hopkins Medicine