Overview and Intended Audience
This section includes guidance regarding:
- Developing structured interview questions.
- Developing structured interview evaluation process.
This content is best suited for residency program staff and/or those who contribute to developing the interview content and establishing the interview processes at your institution. Those who serve as interviewers may benefit from some of this content as well.
Choosing Competencies to Evaluate
It’s important to choose which competencies you will evaluate in your structured interview process before developing interview questions and evaluation content. When selecting competencies, it’s important that they are professionally and/or specialty relevant and represent knowledge areas that residency applicants should be proficient in at the start of their residency (i.e., they should not represent knowledge areas that they will be trained on throughout residency education and are not expected to be proficient in during the selection process). It is also important to ensure the competencies are appropriate to assess in an interview setting. Some competencies, such as those related to technical knowledge, may be best assessed in other parts of the selection process (e.g., educational history, experience, test scores). As a general rule, we recommend selecting three to five competencies to evaluate in your residency selection interview, but propose that programs consider the following factors to help identify how many competencies can feasibly be evaluated:
- Number of interviews conducted per applicant.
- Amount of time allotted for each interview.
- Number of identified competencies the program wishes to evaluate.
- Whether the program wishes to have multiple individuals evaluate the same competencies.
There are several ways to identify potential competencies to evaluate during an interview, including referencing the Accreditation Council for Graduate Medical Education (ACGME) competencies, consulting research relevant for your specific specialty1,2, or any existing competencies your program uses to evaluate residents’ performance.
Behavioral and Situational Questions
There are many types of interview questions. Two of them — behavioral and situational questions — have been widely studied and are considered relatively structured.
- Behavioral questions are based on the premise that past behavior predicts future behavior. They ask applicants to describe what they did in a previous context (typically, in previous jobs, at school, or in volunteer experiences) that provide evidence of important competencies. Past-behavior questions often ask an applicant to describe a specific situation, the behavior or action they took, and the outcome or consequence of that behavior.
Example: Please describe a time when you observed a member of the medical team that you were working with behave in a manner that was inconsistent with an established protocol. Without naming names, please explain what the situation was, what actions you took, and the outcome.
- Situational questions are based on the premise that intentions predict future behavior. They pose hypothetical situations that might occur in residency or as a physician and ask applicants to describe how they would respond in the situations.
Example: I’d like you to imagine that you are on your morning rounds. The chief resident describes a difficult case that you and another PGY-1 worked on earlier in the week and compliments your handling of a difficult situation. In doing so, she gives you sole credit and fails to mention that your colleague played a major role. What would you do?
Refer to the Sample Structured Interview Guide for example behavioral and situational interview questions for two example ACGME competencies.
Maintain a balance between the number of competencies you want to assess, the number of questions needed to assess them, and the amount of time you have available for each interview.
Evaluation-Related Components
A best practice for structured interviews is to use rating scales to evaluate applicants’ responses. In addition to enhancing reliability and validity, rating scales will also increase interviewers’ ability to compare applicants because they were evaluated on a common scale.
There are five key points to note about rating scales:
- A best practice is to develop rating scales for each competency that the interview was designed to assess.
- They can be developed to work with behavioral and situational questions.
- Rating scales used in evaluating interview responses often vary in numbers of scale points, but typically range from three to seven. The chosen number of points corresponds to the range of ability levels or proficiency levels commonly observed among applicants, with wider ranges accommodating more levels within the rating scale.
- Ideally, each point on the rating scale is anchored with behavioral examples that describe each level of proficiency. The behavioral examples on the rating scales should reflect real expectations of each level of performance (in this case, for first-year residents), providing raters with common definitions for each point on the scale. This will both make the rating task easier for raters and help ensure that applicants are being evaluated in a consistent manner.
- Interviewers should be instructed to use the behavioral examples on the rating scale as a general guide for evaluating applicants’ responses.
Refer to the Sample Structured Interview Guide for an example of a behaviorally anchored rating scale for two example ACGME Competencies.
Below we’ve outlined key steps for developing structured interview content:
- Identify key requirements of the position.
- Determine which three to five competencies to target in the interview.
- Develop behavioral or situational questions for each competency (refer to the Sample Structured Interview Guide for examples).
- Invite all parties who interact with medical students to review questions and discuss how applicants would respond.
- Use responses to create draft behavioral examples for each point on the scale (refer to the Sample Structured Interview Guide for examples).
- Review content for cultural appropriateness and understanding, as well as alignment with program goals.
- Document the process and explain how the target competencies and questions were selected.
- Train a pilot group of three to five interviewers on how to use the structured content and rating scale.
- Train all interviewers on how to use the guide and rating scale.
Sample Structured Interview
Sample interview overview
Below is a sample interview. Example structured interview questions and evaluation content are included for reference for two sample competencies: Interpersonal and Communication Skills and Professionalism. Note that while only two competencies are included in the sample guide, your program may evaluate different and/or more competencies in your interviews.
Pre-interview: choosing competencies and questions
Before choosing interview questions, you must first decide which competencies will be assessed in your interviews. This will depend largely on a combination of factors, including how may interviews your residency program conducts, how many competencies you assess, and how long your interviews are. Some competencies (such as those that are more critical for success in your program) can be assessed in multiple interviews, while others may only be assessed in one interview.
For a program that conducts three structured interviews and assesses four competencies per interview, an example competency-interview mapping plan may look like the following:
| Competency | Interview A | Interview B | Interview C |
|---|---|---|---|
| Patient Care and Procedural Skills | X | X | X |
| Practice-based Learning and Improvement | X | X | X |
| Interpersonal and Communication Skills | X | ||
| Professionalism | X | X | |
| Systems-based Practice-based | X | X | X |
Note: The competencies in this table represent ACGME competencies. Medical Knowledge is not included in this table as it is not best assessed in an interview.
During the interview: opening, body, and closing
A note on probing questions
You can use the following probing questions to help you better frame your behavioral and situational interview questions. The questions below are rooted in the situation, task, action, result (STAR) method — a technique for answering behavioral and situational questions in an organized and effective manner. As the interviewer, you will play a key role in ensuring a full STAR answer is given for all competency-based questions. Note that for situational questions, the majority of the applicant’s response will be comprised of action and result since the situation/task is provided to them in the question.
Applicants should focus on one specific scenario in their response rather than describing what they have done in general or describing multiple situations, and part of your role is to keep them focused on one strong situation that addresses the question. You will need to ask as many of these probing questions as necessary to ensure you get a full STAR response from the applicant. For example, the applicant might provide an answer with the situation, task, and action, but fail to elaborate on the result. In that case, you will need to ask the follow-up questions about the outcome or result.
| Probes for Situational Interview Questions | Probes for Behavioral Interview Questions |
|---|---|
|
Situation or Task
Action
Results
|
Situation or Task
Action
Results
|
Interview opening script
Hello, my name is [NAME], and I am the [ROLE] at [INSTITUTION]. Our interview today will be [LENGTH OF TIME] and will include questions that ask you about specific past experiences or how you would handle hypothetical situations. When responding to questions, it is recommended that you tell us a story about what happened; specifically, I would like to hear a short description of the situation or task, the actions you took, and the result of your actions. It is most helpful to focus on one specific example when responding to questions rather than speaking about your experiences in general. You may also discuss group- or team-based examples, but please focus on your specific individual contributions and actions when doing so.
Please let me know if you would like me to repeat any questions during the interview. You may choose to take notes to help collect your thoughts before responding. I will also be taking notes throughout the interview. We will have a few minutes at the end where I can respond to specific questions you have about our program.
Do you have any questions for me before we get started?
Interview questions
Interpersonal and Communication Skills — effective information exchange and teaming with patients, their families, and other health professionals
Choose one question, and ask follow-up probing questions as needed:
- Imagine that you need to communicate difficult news to a patient’s family. Walk me through your approach.
- Tell me about a time when it was difficult to communicate with a patient. What made communication difficult, and how did you navigate the situation?
- Tell me about a time when communicating effectively with other medical professionals helped you improve patient care.
Professionalism — a commitment to carrying out professional responsibilities, adherence to ethical principles, and sensitivity to a diverse patient population
Choose one question, and ask follow-up probing questions as needed:
- Imagine that you are part of a medical team caring for a transgender patient. One of your fellow residents keeps addressing the patient with different pronouns than the patient prefers. What would you do?
- Please provide an example of a time when you had to adjust a treatment for a patient due to their culture or background.
- Imagine that you are part of a medical team caring for a young patient. You learn that one of your fellow residents inadvertently provided a higher dose of a medication than intended to the patient. It will not have any substantial health consequences. What would you do in this situation?
Interview closing
This concludes our interview. Thank you so much for your time and for providing information that allows us to get to know more about you and experiences. [NEXT STEPS TO EXPECT IN SELECTION PROCESS AND GENERAL TIMELINE].