Introduction: The CMMC is an academic body recognized by CONACEM, empowered to certify and recertify physicians specializing in critical medicine. High-fidelity standardized clinical simulation allows the evaluation of aspects that cannot be directly observed during the development of activities in intensive care units, especially for patient safety and the neutrality of the evaluation.

Objective: To evaluate the practical phase of the competency certification exam of 277 applicants through high-fidelity clinical simulation. The examination consisted of six clinical scenarios: high-risk postoperative surgery, sepsis, septic shock, obstetric critical care (mater code), hypovolemic shock, and mechanical ventilation.

Method: The PE developed by the CCMC was carried out on January 12 - 13, 2024 at the CASIC, through five stages, the development of 16 clinical scenarios, checklists, ICTs for the management of checklists and scores, the construction and feeding of the management folder of the exam, the external review of the scenarios and checklists,  the training of facilitators and synod members, the piloting of the exam; Regarding the applicants: development and publication of the call, electronic registration and training to support the exam.

Results: A total of 277 trainees were registered, all of them resident physicians in the last year of the specialty. Of the 277 supporters, 178 (64.3%) men and 99 (35.7%) women, 91% came from public university hospitals, 9% (n=25) from private campuses, Mexico City - metropolitan area with 140 (50.5%) and interior of the Mexican Republic with 137 (40.5%). The states of the Mexican Republic that had the highest number of supporters were Mexico City with 104 (37.5%). The group result had a median of 88 points (79 - 96) (RIQ), the maximum score was 100 and the minimum of 33. The students who obtained a score ³ of 96 were 95 (34.3%), those who obtained a score of 80 - 95 were 111 (40.1%), while the students who had a score of less than 80 were 71, which is equivalent to 25.6%. The best performance was obtained in the mater code evaluation group with a median score of 94.00 (84.75 - 100.00). Conclusion: The evaluation of competencies for certification purposes with the simulation strategy is feasible, feasible both logistically and economically, and extremely useful.

Keywords: evaluation, certification, competencies, simulation, critical medicine

The Mexican Council of Critical Medicine, A.C (CMMC), is an academic body recognized by the National Regulatory Committee of Medical Specialty Councils (CONACEM), empowered to certify and recertify doctors specializing in critical medicine, before the medical community and society in general, to practice the specialty of critical medicine in the Mexican Republic. In the (Certification Regulation. critica.org.mx/cert). The certification process is the mechanism by which the CMMC ensures, through the evaluation of knowledge, skills and attitudes (competencies), that medical specialists in this field meet the quality standards for the practice of the specialty. In our country, the Judicial Weekly of the Federation: In the Supreme Court of Justice of the Federation.¹ The legislation establishes that, in order to practice the medical specialty, it is necessary to pass the certification exam of the corresponding Board Council.

In order to improve the process of evaluating the competencies of intensive care physicians who are certified, the CMMC opted for the high-fidelity standardized clinical simulation technique that allows the evaluation of aspects that cannot be directly observed during the development of activities in intensive care units, above all, for patient safety and the neutrality of the evaluation, avoiding the so-called "schooling" that puts the supporter at a disadvantage. For Cook et al.,² clinical simulation allows for a comprehensive evaluation of the expected performance of the supporters, the areas of strength, the validity of the simulated scenarios and the performance of the tutors; it favors the consolidation of clinical judgment and decision-making in a programmed, immersive, safe, real and reproducible environment; it leads to problem solving through the analysis of cases  in real or everyday situations, through evaluation by trained synod members, approach strategies and the participation of multidisciplinary teams, the result of which brings into play a set of variables that integrate the competence and the significant situations in which the sustainer intervenes, as commented by Oriot et al.³ In summary form to the sustainer,  clinical simulation allows you to analyze in an integral and systematic way the components of the case you are facing, it makes it easier for you to identify principles, laws, norms, techniques and appropriate methodologies that you are going to apply to identify the problem, recognize the cause and solve it. Therefore, learning outcomes are being evaluated as indicated by the Kirkpatrick model. Heydari et al.⁴

Importantly, the first simulators were introduced to the aviation industry in 1920 by pilots in standardized flights without risking life. The first healthcare simulator was used until 1960, when a group of anesthesiologists led by Peter Safar from the United States and Bjørn Lind in Norway developed Resusci Anne, a full-size simulator of a dying victim without a pulse. Subsequently, the Sim One simulator was used with a computer for endotracheal intubation training for anesthesia residents. In 1970, a simulator called Harvey was developed, which was controlled by a computer to teach cardiovascular skills in the detection of heart murmurs. Subsequently, Gaba and collaborators designed a high-fidelity simulator in anesthesia to study the response of the anesthesiologist in critical events simulated in the operating room. Besso et al.⁵

Clinical simulation considers the student as the center of the process, generates continuous training that, by confronting the student daily with the act of solving problems based on reasoning, allows the development of competencies (skills, abilities, knowledge, attitudes), strengthens the integration of basic sciences with clinical sciences, generates formative assessment strategies with specific and pertinent indicators and evaluation instruments,  it favors the recognition of professional degrees and courses with established standards. Clinical simulation is not intended to replace patients or clinical practices, it does not make up for the lack of practice sites, and it will never replace the reality of the lived experience provided by learning about a patient's real case. It does not replace the teacher, it does not replace other well-developed didactic techniques in accordance with the purposes set for the student's training; It is one more didactic strategy and has the quality of optimizing training. In the evaluation, the simulation is on the fourth rung of the Miller pyramid, where the examinee demonstrates how he diagnoses, analyzes and solves the case, from the evaluator's point of view the dimensions are verified: cognitive, technical, integrative, context, rationality, affect, morality and mental habits. In the world there is little evidence regarding the certification of competencies and much less under high-fidelity clinical simulation, so the CMMC has set a precedent by carrying out the practical phase of the Certification Exam under this psychopedagogical and technical paradigm, especially due to the complexity implicit in critical medicine.

To carry out the evaluation of the practical phase of the minimum competencies in critical medicine through high-fidelity clinical simulation, on January 12 and 13, 2024, the CMMC technical examination committee (CTE) carried out five stages:

First stage: development

Clinical scenarios: from March to July 2023, 16 codes of the most frequent pathologies were constructed as a reason for admission and stay in intensive care units, structured based on the single program of medical specialties (PUEM) Graduate, UNAM, support of scientific evidence in critical medicine and evaluation of competencies under the supervision of experts in clinical simulation from CASIC. Annex 1 To facilitate the management of the scenarios codes by synod members and facilitators, decision trees were created (Annex 1).

Checklists: from May to September 2023, the checklists of each code were developed, structured with the criteria and evaluation performances expected (know-know and know-how) by the supporter on a dichotomous scale, based on evidence, constructed in a simple, objective, direct and observable way. We chose the elaboration of this evaluation instrument according to the following characteristics: identification of categories based on the PUEM of Critical Medicine, evaluation indicators that reflect the procedures, skills, behaviors, attitudes and values related to the category to be evaluated, the dichotomous and exclusionary type of assessment scale: present / absent with assignment of a value to each indicator and/or category with which the score was established for approval, in a double-entry table. The allotted resolution time was 10 minutes. The checklists were digitized through the server owned cmmc.databoxmx.com the CMMC in order to optimize their use, facilitate the recognition and scoring of the items by the synod members, the final score was automated due to the limited time of execution of the exam and the number of applicants. In addition, the digitization of the checklists allowed transparency of the evaluation because the score of each item and the final result were verified by the supporters in each of the digital files.

ICTs: from May to September 2023, the CMMC engineering team together with the CTE developed the evaluation module on the server owned cmmc.databox.mx the CMMC, where the checklists were loaded, automating the process of both choosing the list per applicant and the score obtained, giving the applicant the possibility of reviewing both the list and the score and observations obtained during the development of the exam.

Management folder: for audit purposes, the management folder of the exam was fed which contains a privacy and data handling notice, confidentiality agreement of synod members and facilitators, acceptance of recording and use of results by the supporter, work minutes, pilot attendance list, original scenarios codes and their modifications regarding piloting.

Second stage: review

External review: in August 2023, the 16 codes were reviewed by two independent intensive care physicians representing the society.

ICTs: the automation of the checklists was reviewed by CASIC, CTE and CMMC engineering personnel.

Third stage: training

Synod training: in November and December 2023, the CTE and CASIC trained 25 synod representatives from the different university hospital venues, who received the basic simulation course, evaluation of competencies by clinical simulation, use and management of automated checklists.

Training of facilitators: from September to December 2023, the CTE and CASIC trained 20 facilitators, mostly emergency physicians, in mechanical ventilation, hemodynamics, and special situations of each code.

Stage four: piloting

From September to December 2023, each code with its respective checklist was piloted on three occasions by intensive care physicians, representatives of society, and by the synod members themselves.

Fifth stage: evaluation of the quality of the exam

To verify the quality of the exam, the CTE built the satisfaction survey in electronic format through QR in Google forms.

277 applicants registered for the certification exam in the practical phase of the CMMC, all of them resident doctors in the last year of the specialty, who received via electronic video of the facilities of the simulation center both for geographical location and for mobilization within the same center; instructions with the specifications of the type of simulators, their operation and the form of communication with both facilitators and synod members. The examination consisted of six clinical scenarios: high-risk postoperative surgery, sepsis, septic shock, obstetric critical care (mater code), hypovolemic shock, and mechanical ventilation. The sustainers were divided into two groups, each sustainer was assigned a specific schedule to present themselves at CASIC, upon arrival, in a random and blinded manner, the sustainer chose the scenario that he sustained, after the assignment of the scenario the entrance to the five stations was coordinated simultaneously; Each supporter was given the sealed envelope with the brief of the case, they were given 2 minutes to read and, if necessary, doubts. Debrifing was carried out only on the supporters who did not manage to reach the objective during the development of the scenario or difficulties in decision-making were identified. Of the 277 supporters, 178 (64.3%) were men and 99 (35.7%) were women; 252 students took the practical exam, 91% came from public university hospitals, 9% (n=25) from private campuses. The applicants were divided into two groups according to their origin: Mexico City - metropolitan area with 140 (50.5%) and interior of the Mexican Republic with 137 (40.5%). The states of the Mexican Republic that had the highest number of supporters were Mexico City with 104 (37.5%), State of Mexico 36 (13%) and Guadalajara 20 (7.2%); while the states where there were fewer supporters were Michoacán with one support (0.4%), Chihuahua and Hidalgo with 2 (0.7%) supporters each and Oaxaca with 3 (1.1%) supporters.

During the simulation certification exam, 5 scenarios were set up: sepsis where 95 supporters (34.3%) were evaluated, mater code with 48 supporters (17.3%), high-risk surgery with 45 students (16.2%), hypovolemic shock with 45 students (16.2%) and finally in the mechanical ventilation scenario 44 students (15.9%) of the total participated. The group result had a median of 88 points (79 - 96) (RIQ), the maximum score was 100 and the minimum of 33. The students who obtained a score ³ of 96 were 95 (34.3%), those who obtained a score of 80 - 95 were 111 (40.1%), while the students who had a score of less than 80 were 71, which is equivalent to 25.6%. The best performance was obtained in the mater code evaluation group with a median score of 94.00 (84.75 - 100.00) (Table 1, Figure 1). The best performance by state of the republic by scenario: in the case of sepsis was Aguascalientes, followed by Guanajuato and Sonora, while the worst performance was obtained by Monterrey, Tamaulipas and Guadalajara. In the mater code scenario, the best performance was obtained by Veracruz, Monterrey and Guanajuato, while the lowest performance was obtained by Yucatán, Tamaulipas and Sonora. For the high-risk surgery scenario, the best performance was for the sustainers from Veracruz, Guadalajara and Guanajuato, while the lowest yield was for the sustainers from Tamaulipas, Mexico City and Yucatán; for hypovolemic shock, the best scores were obtained by Aguascalientes, Yucatán and the State of Mexico, while the lowest performance was obtained by Hidalgo, Guanajuato and Mexico City, finally, the mechanical ventilation scenario had the best performance obtained by supporters from the state of Yucatán, Querétaro and Guanajuato, while the lowest was obtained by Puebla, State of Mexico and Monterrey (Table 2).

Figure 1 Qualifications by Scenario.

A difference in scores was obtained between the groups from the interior of the republic and Mexico City - metropolitan area, as well as between private and public venues for each scenario, finding that there was only a significant difference in the mechanical ventilation scenario in supporters from the interior of the republic and Mexico City - metropolitan area with a p = 0.014 and in the hypovolemic shock scenario group in private and public venues, a difference was determined with  p = 0.0029 (Table 3). Regarding the quality of the evaluation, 94.4% (250) received guidance and resolution of doubts, 100% (270) agreed with the adequacy of the CASIC facilities, 98.8% (270) thought that the capacity of the facilitators was optimal, 98% (250) reported that the scenario was immersive, 89.2% (250) thought that the scenarios were clear and understandable; finally, 92.7% (250) agreed with the simulation evaluation. Between 5 and 20% of the supporters said that they required more clinical data to solve the case, more emphasis on the logistics of entering and leaving the scenarios and more time for debriefing.

To assess the competencies of critical care medicine physicians, the need for more advanced evaluations has been created, in order to obtain higher-level results that reflect the ability of physicians to demonstrate their skills, as indicated by Feldman et al.⁶ For Cook et al.,² assessment is a central and under-recognized component in medical care; we continually make judgments and decisions both in daily practice and with students who are given different types of assessment, including test scores, however, as education evolves we are currently immersed in the new paradigm:  competencies, where domains and learning objectives are evaluated. For Besso et al.,⁵ physicians educated under this paradigm must have continuous training that includes a structured process that progressively transfers increasing levels of responsibility that includes the practical aspects of care with the aim of providing adequate training and experience in the administrative and management functions of the Intensive Care Unit.

The use of techniques such as simulation-based team building as an educational tool is increasing in recent decades, as mentioned by L'Her et al.⁷ Healthcare professionals prefer simulation training because it strengthens knowledge, improves skills, and verifies behaviors. The objective of simulation-based training is to increase the quality of treatment and patient safety, optimizing diagnoses and interventions in which time is critical with an impact on morbidity and mortality.⁵ For Boyle et al.,⁸ I determine that simulation evaluation is very useful to determine the willingness to assume the role of intensivist, verify decision-making and acquired competencies.  Nazar, Bloch, & Fuentes.⁹ report that simulation is underused both in Latin America, specifically Chile, and in countries such as the United States, Australia and Canada, where there is more experience but few publications on the subject.  Regarding the usefulness of clinical simulation for certification purposes, The American Board of Anesthesiology¹⁰ reports that France presents simulation for certification purposes. Gaugler and Rudolph¹¹ report that in the United States they contemplate the use of this resource in the process. Thus, the CMMC is venturing and positioning itself at the forefront in the evaluation of competencies of postgraduates in critical medicine in Mexico.

As can be seen in the method detailed above, the methodology used was similar to that of the Francophone Society of Simulation in Health (Societé Francophone de Simulation en Santé).¹² where competencies were clearly defined, knowing how to act in a professional situation, as a first instance, identifying both internal resources such as behavior, knowledge and skills, as well as external resources: team and collaborators. The number of competencies to be assessed, measured through checklists structures according to Henricksen, Altenburg and Reeder. ¹³were standardized, valid and reproducible as also indicated by Messick;¹⁴ whose content was developed by an expert in evaluation and two experts in critical medicine, reviewed by two experts in critical medicine external to the team who verified the relevance and co-ordination of the items through Cohen's Kappa, discrimination and coherence; together the experts determined the minimum of the percentage.

From the results obtained we can say that the group median exceeded 80 of qualification, however the qualification interval was wide with a minimum of 33, the bulk of the evaluated population remained in the interval of 80 – 95; the scenario with the highest number of sustainers was sepsis because it is the pathology that leads the admission to the intensive care unit in our country and worldwide as a result of the its impact on patient morbidity and mortality,¹⁵ the most complex scenario was high-risk surgery, whose cases to be resolved were mainly heart surgery. The scenarios were chosen as Buléon et al.¹⁶ comment, for their representativeness in professional life and potential impact on the patient. The duration of each scenario was 10 minutes, the applicant was given the brief and the precise list of competencies to be evaluated; each scenario was piloted on more than four occasions, all the participants were videotaped; due to the number of students and the time of use assigned per day of the CASIC, it was only possible to evaluate one scenario per student.  As in the study by Rudolph et al..¹⁷ A situation to highlight is that the critical medicine training sites continue to be centralized in Mexico City and the metropolitan area with more than 50% of the supporters, such centralization does not result in better performance, sites such as Guadalajara, Monterrey and Guanajuato showed similar or superior performance to the center of the country, especially in the performance of hypovolemic shock and mechanical ventilation scenarios with a statistically significant difference.

The synod members were trained in the evaluation of competencies under clinical simulation by the CASIC, the use of digitized checklists, the simulators that would be used in the corresponding scenario and in the logistics of the day of the exam also conotated as D-day. In the trainings, emphasis was placed on potential errors, effects and biases, such as the homogenization error that is verified with the tendency to score very well and very badly, generating problems for the discrimination of the lists; the Halo effect as used  by Uzan,¹⁸ which manifests itself when seeing everything good or bad in the performance of a supporter; the time effect, attributed to the bias of verifying a good or bad performance early or late between sessions; the clemecia bias that consists of resisting giving bad scores. Due to the impact produced by both the consequences of the result of the evaluation and the support of the exam, we have the support of two masters in Psychology to safeguard the psychological safety of doctors, attributed to the anguish, anxiety and stress experienced by the student in the 36 hours prior to the exam, as referred to by Lateef¹⁹ psychological safety is the ability to express oneself without fear of negative consequences or the feedback that their speech, comment or action can generate, provides cognitive, physical and emotional freedom. Also as indicated by Kang and Min.²⁰ One situation that the CMMC sought was the transparency of the objectives and methods of the exam, which in addition to giving legal certainty to the exam impacted the psychological safety of the applicant.

The construction of the review, logistics and implementation was a challenge of the UNFCCC and each stage meant a series of obstacles both foreseeable and not visible to overcome, especially in the face of the lack of information, the lack of cooperation to share experiences and results.

With checklists, only inter-observer and Cronbach's alpha agreement can be made in the area of statistical analysis.

Competency assessment for certification purposes with the simulation strategy is feasible, logistically feasible and economically, and extremely useful.

NACC: coordination of clinical scenario construction, pilot guidance, and evaluation logistics; statistical analysis, writing and review of the article

BEHM: statistical analysis, writing and review of the article.

ACA: coordination of clinical scenario construction, pilot guidance, and evaluation logistics; writing and review of the article.

JCSS: clinical scenario construction, pilot guidance, and evaluation logistics, review of the article.

RMG: construction of checklists, pilot guidance, and evaluation logistics, review of the article.

NPM: clinical scenario piloting and evaluation logistics.

IHSG: clinical scenario piloting and evaluation logistics, development of the evaluation satisfaction survey, statistical analysis of the survey; review of the article.

None.

1. Suprema Corte de Justicia de la Federación. Semanario Judicial de la Federación. 2025.
2. Cook DA, Brydges R, Ginsburg S, Hatala R. A contemporary approach to validity arguments: a practical guide to Kane’s framework. Med Educ. 2015;49(6):560–575.
3. Oriot D, Darrieux E, Boureau–Voultoury A, Ragot S, et al. Validation of a performance assessment scale for simulated intraosseous access. Simul Healthc. 2012;7(3):171–175.
4. Heydari MR, Taghva F, Amini M, et al. Using Kirkpatrick’s model to measure the effect of a new teaching and learning methods workshop for health care staff. BMC Res Notes. 2019;12(1):388.
5. Besso J, Bhagwanjee S, Takezawa J, et al. A global view of education and training in critical care medicine. Crit Care Clin. 2006;22(3):539–546.
6. Feldman M, Lazzara EH, Vanderbilt AA, et al. Rater training to support high–stakes simulation–based assessments. J Contin Educ Health Prof. 2012;32(4):279–286.
7. L’Her E, Geeraerts T, Desclefs J, et al. Simulation–based teaching in critical care, anaesthesia and emergency medicine. Anaesth Crit Care Pain Med. 2020;39(2):311–326.
8. Boyle WA, Murray DJ, Beyatte MB, et al. Simulation–based assessment of critical care “front–line” providers. Crit Care Med. 2018;46(6):e516–e522.
9. Nazar C, Bloch N, Fuentes R. La simulación como herramienta de evaluación de competencias y certificación. Simul Clin. 2019;1(2):104–110.
10. The American Board of Anesthesiology. Maintenance of Certification in Anesthesiology (MOCA) 2.0. 2025.
11. Gaugler B, Rudolph A. The influence of assessee performance variation on assessors’ judgments. Pers Psychol. 1992;45(1):1–22.
12. Downing SM. Validity: on the meaningful interpretation of assessment data. Med Educ. 2003;37(9):830–837.
13. Henricksen JW, Altenburg C, Reeder RW. Operationalizing healthcare simulation psychological safety. Simul Healthc. 2017;12(5):289–297.
14. Messick S. Educational Measurement. 3rd ed. New York, NY: American Council on Education and Macmillan; 1989:14–103.
15. Estadísticas de Defunciones Registradas (EDR). 2025.
16. Buléon C, Mattatia L, Minehart RD, et al. Simulation–based summative assessment in healthcare: an overview of key principles for practice. Adv Simul (Lond). 2022;7(1):38.
17. Rudolph JW, Simon R, Raemer DB, et al. Debriefing as formative assessment: closing performance gaps in medical education. Acad Emerg Med. 2008;15(11):1010–1016.
18. Uzan S. Mission de recertification des médecins: exercer une médecine de qualité. Ministère de l’Enseignement supérieur, de la Recherche et de l’Innovation. 2025.
19. Lateef F. Maximizing learning and creativity: understanding psychological safety in simulation–based learning. J Emerg Trauma Shock. 2020;13(1):5–8.
20. Kang SJ, Min HY. Psychological safety in nursing simulation. Nurse Educ. 2019;44(2):E6–E9.