Monitor and Evaluate

Implementing AI ethically in selection requires ongoing vigilance through regular monitoring and evaluation. Just as with any element of the selection process, it is essential to provide ongoing evidence that demonstrates the validity, reliability, and fairness of AI tools and systems. This persistent oversight ensures that AI implementations stay aligned with the institution’s goals and ethical standards and helps manage legal and data security risks. 

From principle to practice:

• Establish standards for ongoing evaluation. Implement standards for classification metrics, interpretability, and data/concept drift.1 Additionally, monitor both user and applicant reactions to ensure the system is perceived as effective, fair, understandable, and aligned with institutional goals for the tool. This approach will help to identify areas for improvement and promote compliance with ethical standards.
• Monitor and adjust after each cycle. At the conclusion of each application cycle, conduct a thorough review of the AI system and its outputs. Assess whether and when adjustments are needed in the model, its application, or in the training provided to those who operate it. This regular monitoring ensures that the AI tool continues to align with institutional goals and adapts to any changes in the broader context of selection.
• Remain responsive. Keep the AI system current and relevant by adapting to significant changes in institutional curriculum and goals and in the applicant pool. Being proactive will help to maintain the system’s effectiveness and alignment with each institution’s evolving needs.
• Document development and implementation. Create a technical report or other documentation describing the steps and decisions involved in the development and implementation of AI systems. The report documentation should align with the Liaison Committee on Medical Education (LCME^®) and other relevant requirements to support thorough evaluations and audits. Additionally, develop a standard operating procedure in accordance with the LCME requirements to ensure clarity and consistency for admissions officers.