Educating the pediatrician of the 21st century: defining and implementing a competency-based system
Carol Carraccio
The Accreditation Council for Graduate Medical Education (ACGME) has partnered with the American Board of Medical Specialties (ABMS) in responding to public concerns regarding accountability in health care and medical education through the “Outcomes Project.” (1) As a result, all graduate-level training programs have been mandated to change the infrastructure of the training process to one that is competency based. Although the changes in graduate medical education have received the most focused attention, generalization to undergraduate education of medical students and to the certification process of practicing physicians is also underway. The purpose of this article is twofold: 1) to update pediatricians about competency-based education and its impact on the pediatric community at large and 2) to describe the first stages of the multistep process to implement a competency-based system of education in a university setting.
DEFINITIONS
In an attempt to define a competency-based system of education, one must first come to grips with a definition of “competency.” Review of the literature shows a number of definitions that, when synthesized and simplified, describe “competency” as a complex set of behaviors built on the components of knowledge, skills, and attitudes. (2-9) In contrast “competence” refers to one’s ability to perform a task. The focus on competencies differs significantly from our current structure/process system of education, in which the focus of training is on knowledge acquisition and the process is teacher centered. In a competency-based system, the focus is on outcome, which is the application of the knowledge, and the process is learner centered with input from a mentor.
The ACGME named the following 6 competencies that all graduate-level trainees must attain by the completion of formal training: patient care, medical knowledge, interpersonal and communication skills, professionalism, practice-based learning and improvement, and systems-based practice. The former 4 competencies are traditionally areas of focus within graduate medical education with the emphasis on professionalism being more recent. The latter 2 domains of competence are less familiar but critically important as one reflects on the environment in which medical care is practiced in the 21st century. The essence of practice-based learning and improvement is the commitment to practice improvement by 1) designing and evaluating interventions to address identified problems, 2) the use of technology in the acquisition and application of “best evidence,” and 3) the commitment to life-long learning. Competence in systems-based practice requires us to demonstrate an “awareness of and responsiveness to the larger context and system of health care and to effectively call on system resources to provide care that is of optimal value.” (1) It includes one’s ability to partner with other professionals to navigate and advocate for patients within the context of the health care delivery system. The practitioner is challenged to balance cost with quality. Also included is a focus on system errors and one’s ability to improve the system by impacting these errors.
The sequel to identifying the broad competencies was to define the elements that comprise them. The ACGME and ABMS laid the groundwork for this step but garnered the help of 24 teams, 1 for each of the 24 ABMS specialties. Each team was comprised of a representative from the specialty board, program directors association, residency review committee, and a resident, hence earning the name “specialty quad.” Thus the initial draft of competencies was refined and specialty specific language and elements were incorporated (see Table 1). (10) At this juncture, the education community was challenged to continue the effort initiated by the ACGME by developing best-practice models to bring the Outcomes Project to fruition.
LAYING THE GROUNDWORK FOR CHANGE
Accepting this challenge, the program directors from the departments of pediatrics, internal medicine, and family medicine applied for and received an academic administrative unit grant from the Health Resources and Services Administration to develop a collaborative educational initiative that would move us in the direction of a competency-based system of education. The program directors, clerkship directors, and ambulatory clinic directors, as well as a medical educator, became members of the work group that brought this initiative to fruition.
In an effort to understand the logistics and process for implementation of a competency-based system of education, we performed a literature search on competency-based education dating back to the beginning of both the ERIC and MEDLINE systems. (2) Review of the literature on the development of competency-based curricula and evaluation outlines a 4-step process. The 4 steps include 1) competency identification, 2) determination of competency components and performance levels, 3) evaluation of competence, and 4) an overall assessment of the process itself. The ACGME took the first step with the initiation of the Outcomes Project in 1999 by defining the 6 competencies. (1) The remainder of this article describes how we accomplished step 2, determination of competency components and performance levels.
As we began this undertaking, the working group met to develop the framework for this project. Our first major task was to mesh discipline-specific curricula, the traditional goals and objectives, with the 6 competencies. Consensus was that we develop benchmarks for the 5 competencies that crossed disciplinary lines as well as the 2 components of the medical-knowledge competency that dealt with knowledge acquisition and application. For the third component of the medical-knowledge competency, dealing with specialty-specific content in the form of goals and objectives, faculty within each specialty developed their own curriculum. Our framework then embodies the traditional curriculum as 1 component of the medical-knowledge competency.
STEP 2: DETERMINATION OF COMPETENCY COMPONENTS AND PERFORMANCE LEVELS
Phase 1: Defining Benchmarks
Step 2 in the process of shifting to a competency-based system is to further define the elements of the competencies by determining the measurable behavioral objectives that comprise the elements of the competency. These behavioral objectives are referred to as benchmarks or performance indicators. For example, the patient care competency includes a number of elements that were defined by the ACGME in conjunction with the specialty quad. One of these elements is “gather essential and accurate information about the patient.” Each element of the competencies was assigned to a program director or associate program director with the charge to develop a draft of measurable benchmarks or performance indicators for that element. (See the Appendix for an excerpt. The full grid can be found on the Association of Pediatric Program Directors [APPD] web site at www.appd.org under resources for competency evaluation, evaluation portfolio). The working group reviewed the drafts in an effort to minimize errors of omission or commission and to ensure that benchmarks were assigned to the most appropriate competency, especially in areas of overlap. Attention also was focused on the specific use of behavioral verbs to define the benchmarks as measurable tasks. This process was reiterated until there was consensus from the group. The entire first phase of the project took several months to complete. Subsequently, the clerkship directors developed a parallel document to address the same domains of competence for undergraduate trainees. The literature describing a similar process for preventive medicine was particularly helpful in guiding us through this exercise. (11-14)
Phase 2: Defining the Thresholds
The second phase of step 2 involved setting thresholds that demonstrate competence. We started by defining the types of thresholds to be used in determining whether competence for a given benchmark was achieved. In a focus-group session of the Program Director’s Committee of the American Board of Pediatrics (ABP), 3 types of thresholds were suggested. The first involved the percentage of time that a learner accomplished a task. An example of a benchmark that would fit with this type of threshold is “interviews patients with an appreciation of their developmental level and/or age,” where one may expect a competent junior resident to perform this task 25% to 50% of the time while expecting a competent senior resident to demonstrate this skill >75% of the time. The second type is based on one’s ability to complete the task based on acuity / complexity of patient problems, with more junior learners demonstrating competence in a skill with routine patients but more-senior learners needing to demonstrate competence in the skill regardless of patient acuity or complexity. An example of such a benchmark would be “performs a detailed and accurate physical examination.” The third type of threshold involved dichotomous categories of behavior (eg, does or does not demonstrate a behavior). This would be the most-frequent type of threshold for the professionalism competency. Two of the authors (C.C. and R.E.) applied 1 of the 3 specific types of thresholds to each benchmark listed under each of the 6 domains of competence. These designations were brought back to the multidisciplinary group for approval. This preliminary work was presented to the Residency Review Committee for Pediatrics in October 2001 and modified further based on input from committee members.
Phase 3: Setting Thresholds for Competence
The third phase of step 2 required the establishment of consensus regarding the threshold for competence at each level of training for each specific benchmark. To accomplish this task, the document was sent to the membership of the APPD along with a cover letter explaining the content and process of the draft. Each program director was asked to assign the most-appropriate threshold for each level of training for each benchmark at the end of that postgraduate year (PGY). Responses were sought for residents at the following PGYs of training: PGY-0.5 (half-way through year 1), PGY-1, PGY-2, and PGY-3 (the end of years 1, 2, and 3, respectively). Follow-up to the initial mailing was done through several e-mail communications. All surveys were anonymous.
Survey results were analyzed by using the Statistical Package for the Social Sciences. Frequencies of responses were determined for each benchmark at each of the specified levels of training. A consensus response was considered one in which >50% of program directors agreed on the threshold for the level of training for the specific benchmark. In cases for which there was not a clear-cut consensus, the minimum threshold was chosen. For example, if 45% of program directors felt that a PGY-1 should be able to perform a task for routine patients to have achieved competence for that benchmark and 40% felt that a PGY-1 should be able to perform a task on most patients, the minimum threshold (ie, demonstrating the task for routine patients) was considered to be the consensus for that benchmark at the PGY-1 level of training.
Of the 208 programs in categorical pediatrics accredited by the ACGME, 202 are members of the APPD. Of these 202 programs, 81 completed the survey, for a response rate of 40%. Although some controversy existed for thresholds at time PGY-0.5 and PGY-1 levels, 95% and 100% agreement were reached for the PGY-2 and PGY-3 levels, respectively. Although the 40% response rate is a limitation of these data, the fact that clear consensus was achieved for 95% of benchmarks at the PGY-2 level and 100% of benchmarks at the PGY-3 level adds to the face validity of the threshold. Additionally, for all benchmarks the consensus threshold increased as level of experience of the trainee increased, thus lending discriminant validity to the instrument. Examples of the consensus thresholds for each benchmark at each level of training are shown in the Appendix.
STEP 3: EVALUATING COMPETENCE
Steps 1 and 2 of the shift to competency-based education are challenging but pale in comparison to step 3, which requires the evaluation of competence. The process for evaluating competence differs from the typical approach to evaluation that has been used in the current medical education system. Most programs use global evaluations that are generic in nature, with a single evaluation form serving as the assessment tool for a number of different clinical experiences and settings. The components of the evaluation are often a proxy for the task being evaluated. The evaluator’s response is typically based on general interactions with the learner over a prescribed period of time. The evaluator may never have observed the learner performing a given task directly but rather extrapolated an assessment based on discussions with the learner regarding patient care activities. In addition, in the current system, learners are compared with each other, a process known as norm-referenced assessment. In contrast, competence is evaluated by using criterion-referenced assessment; that is, the learner must meet a predefined threshold to be considered competent. (15) Thus, if one is to evaluate competence in patient care, one must observe directly the benchmarks that comprise ability in patient care. A directly observed history and physical or an interview with a standardized patient provide 2 potential ways of assessing competence in patient care. The individual’s performance of a history and physical is not measured by comparison to the other members of the team. Rather, each resident must simply meet the predefined standard for performing the history and physical examination. Implicit in judging a learner’s ability to meet thresholds is the teacher’s ongoing feedback to the learner as part of the educational process to guide the learner in meeting standards that have been set. A competency-based system of education emphasizes formative or ongoing feedback rather than summative feedback given at the end of the clinical experience or rotation.
The diversity of the domains of competence that have been defined by the ACGME makes the evaluation process a particular challenge. To meet this challenge, we must identify and develop a variety of tools to assess competence, the tools designed to measure competence in systems-based practice being quite different from those needed to evaluate professionalism. In fact, the relative newness of the concepts of competency in systems-based practice and practice-based learning and improvement underscore the need for the medical education community to develop new assessment tools. In addition, we are challenged to begin to study the reliability and validity of any new assessment measures. In response to this challenge we are in the process of developing a web-based evaluation portfolio to assess the diverse range of competencies required of graduate medical trainees.
STEP 4: EVALUATING THE OUTCOME
Step 4 of the process is the evaluation of the outcome of the shift to a competency-based system. This step requires us to determine whether the implementation of a competency-based system of education results in “better” physicians and, ultimately, improved patient outcomes. Only through improved patient outcomes can we establish the ultimate effectiveness of the ACGME Outcomes Project. This step will require the input and collaboration of national organizations such as the ACGME, ABMS, and the National Initiative for Children’s Healthcare Quality, as well as others.
GENERALIZING THE COMPETENCIES TO MEDICAL STUDENTS AND PRACTITIONERS
Although the changes that are occurring in graduate education have received the greatest press, similar changes are occurring in both medical school and continuing medical education. The Association of American Medical Colleges supports a shift to competency-based education for medical students. Aligning student and resident competencies allows us to begin to address education along the continuum. The key here is pairing leadership in undergraduate and graduate education to ensure that outcome competencies from medical school match expected incoming competencies for residency training.
The practicing physician is now being called on to demonstrate competence through continuous professional development as a requirement for maintaining certification. At the level of board certification, the member boards of ABMS, of which the ABP is one, are shifting away from “recertification” for practicing physicians to “maintenance of certification.” The essence of this shift parallels what is happening at the undergraduate and graduate levels of education. The practicing physician will need to demonstrate not only current knowledge but also evidence of professional standing, a commitment to life-long learning and satisfactory performance in practice. For pediatricians, there will be 4 steps to maintaining one’s general pediatric certification. (16) Step 1 requires pediatricians to maintain valid medical licenses in all states in which they hold a license. Step 2 is demonstrating a commitment to life-long learning through the Knowledge Self-Assessment and Decision Skills Self-Assessment. The Knowledge Self-Assessment is an on-line activity that the ABP will make available on its web site (www.abp.org) in late 2004. This requirement also can be accomplished by completing the PREP Self-Assessment offered by the American Academy of Pediatrics. The Decision Skills Self-Assessment is offered only by the ABP and will require attaining a passing score. Step 3 will involve successfully completing a secure examination at local testing centers. Step 4 will involve 2 practice improvement activities: 1) peer/patient surveys and 2) a performance in practice activity. The peer/patient process is still in the developmental stages and will not be required for several years. Completing an ABP-approved program for quality improvement can fulfill the performance in practice activity. The American Academy of Pediatrics has designed an on-line program called eQIPP (Education in Quality Improvement for Pediatric Practice) that meets the ABP’s standards for a performance in practice activity. The phase-in of the new maintenance of certification process will be complete by 2010. Diplomates with certificates that expire before 2010 will need to complete the first (medical license) and third (examination) steps of the process; diplomates with certificates issued from 2010 onward will need to complete all 4 components successfully.
LESSONS LEARNED
There were several lessons learned from the process of developing these benchmarks and thresholds. The first lesson involved the identification of key faculty for the working group. The establishment of a multidisciplinary group of faculty involving individual stakeholders in the medical education process, from undergraduate through graduate training, enriches the process, garners buy-in, and improves the outcome by virtue of the collaborative process. In particular, the input of the group as a whole significantly decreases the number of errors of omission that each faculty member commits on his/her own. The input of a medical educator is also extremely helpful, particularly in providing an understanding of the lexicon of competency-based education. The process requires a champion who is passionate about the project. This individual will be pivotal in motivating the group, particularly at the inevitable times of inertia.
The second lesson learned was that establishing the framework for relating goals and objectives to competencies is a major hurdle. Reaching this milestone allowed us to develop the infrastructure for the educational program. The 6 competencies became the foundation, and the elements of the competencies were defined further by benchmarks. The medical-knowledge competency embraces acquisition and application of knowledge as well as discipline-specific content in the form of a curriculum. The latter is framed in traditional goals and objectives. Each department would write its specific curricula, but the rewards of this collaborative effort inspired us to continue work group efforts on curricula that crossed disciplinary lines such as substance abuse, nutrition, and complementary medicine, to name a few.
The third lesson learned was the difficulty in getting responses by survey mailings. We established thresholds for the benchmarks based on only a moderate survey response from the APPD membership. The response rate likely was affected by the length of the survey and potential lack of understanding regarding how consensus on these benchmarks and thresholds would help the pediatric community take the next step in the process. Despite the moderate response rate, clear consensus on thresholds for PGY-2s and PGY-3s was encouraging. Moving forward we will need to test the hypothesis that the expected resident performance as reported in this survey is consistent with the actual performance of our residents.
IMPACT ON THE PEDIATRIC COMMUNITY
The magnitude of these changes in education has broad implications for the pediatric community. All pediatricians who hold a time-limited certificate from the ABP will need to “maintain certification.” In addition, practicing physicians and academic faculty who contribute to the education of medical students and residents will need to provide more direct observation of the learner performing authentic tasks, along with formative feedback to the learner about performance. Finally, those pediatricians who have assumed roles in educational leadership will have the added responsibility of 1) developing curricula to address the required competencies and 2) creating assessment tools to demonstrate to accrediting agencies and the general public that our graduates have met performance standards for the 6 ACGME competencies. Our hope in initiating the second step in this process was to provide the pediatric community with a foundation on which to build an evaluation system that one can use to assess competence. The major challenges in the third step will be 1) creating tools that measure competence of the trainee in performing the tasks of the practicing physician that are both cost-effective and practical and 2) testing the reliability and validity of any of the new development-assessment tools. We are currently in the process of creating a web-based evaluation portfolio that will address the benchmarks for the 6 ACGME competencies. Only when we have addressed the third step can we systematically study the pivotal question of whether this shift to a competency-based system of education makes a “difference.” This final step will require a national effort combined with a national consensus on defining “difference.” We speculate that various organizations and interest groups will define “difference” based on the issues critical to the organization itself. Thus, measurable differences in job satisfaction, cost of care, etc may result. We may say that we have trained “better” physicians based on higher board scores or decrease in the number of medical errors; however, the “real difference” can be measured only in the quality of care delivered to our patients.
APPENDIX: EXCERPT OF BENCHMARKS AND THRESHOLDS FOR COMPETENCE IN PATIENT CARE
Residents must be able to provide family-centered patient care that is developmentally and age appropriate, compassionate, and effective for the treatment of health problems and the promotion of health.
TABLE 1. Pediatrics General Competencies and Synopsis of Competency
Elements
Competencies Elements of Competency:
Patient care
Residents must be able to * Gather essential and
provide family-centered patient accurate information
care that is developmentally * Make informed diagnostic
and age appropriate, and therapeutic decisions
compassionate, and effective * Carry out patient-
for the treatment of health management plans
problems and the promotion * Prescribe and perform
of health competently all medical
procedures
* Counsel patients and
families
* Provide effective health
maintenance and
anticipatory guidance
* Use information technology
to optimize patient care
Medical knowledge
Residents must demonstrate * Demonstrate an
knowledge about established and investigatory and analytic
evolving biomedical, clinical, approach to clinical
and cognate (eg, epidemiological problem solving and
and social-behavioral) sciences knowledge acquisition
and the application of this * Know, apply, and teach the
knowledge to patient care and basic and clinically
the education of others supportive sciences
Practice-based learning and improvement
Residents must be able to investigate * Analyze practice experience
and evaluate their patient care and perform practice-based
practices, appraise and assimilate improvement activities
scientific evidence, and improve * Locate, appraise, and
their patient care practices assimilate evidence from
scientific studies related
to one’s patients health
problems
* Obtain and use information
about one’s own population
of patients and the larger
population from which the
patients are drawn
* Apply knowledge of study
designs and statistical
methods to the appraisal of
clinical studies
* Use information technology,
peer review, and self-
assessment to promote
life-long learning
* Facilitate the learning of
students and other health
care professionals
Professionalism
Residents must be able to demonstrate * Communicate effectively to
interpersonal and communication create and sustain a
skills that result in effective therapeutic relationship
information exchange and teaming with patients and families
with patients, their patient’s * Work effectively with
families, and professional associates others as a member or
leader of a health care
team or other professional
group
Systems-based practice
Residents must demonstrate a * Demonstrate respect,
commitment to carrying out compassion, and integrity;
professional responsibilities, a responsiveness to the
adherence to ethical principles, needs of patients and
and sensitivity to a diverse society that supercedes
patient population self-interest; accounta-
bility to patients,
society, and the
profession; and a
commitment to excellence
and on-going professional
development
* Demonstrate a commitment to
ethical principles
pertaining to provision or
withholding of clinical
care, confidentiality of
patient information,
informed consent, and
business practices
* Demonstrate sensitivity and
responsiveness to a
diverse patient population
Systems-based practice
Residents must demonstrate an * Know how types of medical
awareness of and responsiveness to practice and delivery
the larger context and system of systems differ from one
health care and the ability to another
effectively call on system resources * Practice cost-effective
to provide care that is of optimal health care and resource
value allocation that do not
compromise quality of care
* Advocate for quality
patient care and assist
patients in dealing with
system complexities
* Partner with health care
managers and health care
providers to assess,
coordinate, and improve
health care
* Understand the reciprocal
impact of personal
professional practice,
health care teams, and the
health care organization
on the community / society
TABLE 2. Patients With CHD
Defect No. of Patients
Anomalous left coronary artery 1
Anomalous pulmonary venous 1
connection
Aortic regurgitation 2
Aortic stenosis 6
Atrial septal defect 8
Atrioventricular septal defect 1
Cardiac tumor 1
Cardiomyopathy 3
Coarctation of the aorta 3
Double inlet left ventricle 1
Ebstein’s anomaly of the tricuspid valve 7
Hypoplastic left heart syndrome 2
Mitral regurgitation 4
Mural valve prolapse 26
Patent ductus arteriosis 3
Pulmonary atresia 1
Pulmonary stenosis 4
Tetrology of Fallot 5
D-transposition of the great arteries 4
L-transposition of the great arteries 1
Tricuspid atresia 2
Ventricular septal defect 11
Total 97
ACKNOWLEDGMENTS
This work was supported in part by a grant from the Health Resources and Services Administration of the Bureau of Health Professions.
We thank Drs Robert Perelman, Gall McGuinness, and Hazem Ham for their advice in preparing this manuscript.
REFERENCES
(1.) Accreditation Council for Graduate Medical Education. Outcomes Project. Available at: www.acgme.org/Outcome. Accessed January 2, 2003
(2.) Carraccio C, Wolfsthal SD, Englander R, Ferentz K, Martin C. Shifting paradigms: from Flexner to competencies. Acad Med. 2002;77:361-367
(3.) Bell CG. Role vs. entry-level competencies in competency-based education. J Am Diet Assoc. 1976;69:133-137
(4.) Brown TC, McCleary LE, Stenchever MA, Poulson AM. A competency-based educational approach to reproductive biology. Am J Obstet Gynecol. 1973;116:1036-1043
(5.) Bell HS, Kozakowski SM, Winter RO. Competency-based education in family practice. Faro Med. 1997;29:701-704
(6.) Grussing PG. Education and practice: is competency-based education closing the gap? Am J Pharm Educ. 1984;48:117-124
(7.) Hager P, Gonczi A. Competency-based standards: a boon for continuing professional education? Stud Contin Educ. 1991;13:24-40
(8.) Rinke WJ. Competency-based education. J Am Diet Assoc. 1980;76: 247-252
(9.) Spady WG. Competency-based education: a bandwagon in search of a definition. Educ Res. 1977;6:9-14
(10.) Accreditation Council for Graduate Medical Education. Pediatrics general competencies. Available at: www.acgme.org/RRC_PedDocs/ ped.GenComp.pdf. Accessed August 22, 2003
(11.) Lane DS, Ross V. Consensus on core competencies for preventive medicine residents. Am J Prey Med. 1994;10:52-55
(12.) Lane DS, Ross V. The importance of defining physician’s competencies: lessons from preventive medicine. Acad Med. 1994;69:972-974
(13.) Lane DS, Ross V, Purkinson MD, Chen DW. Performance indicators for assessing competencies of preventive medicine residents. Am J Prey Med. 1995;11:1-8
(14.) Pololi LH, Coletta EM, Kern DG, et al. Developing a competency-based preventive medicine curriculum for medical schools. Am J Prey Med. 1994;10:240-244
(15.) Turnbull J. What is … normative versus criterion referenced assessment. Med Teach. 1989;11:145-150
(16.) Stockman JA, Ham H. Maintenance of certification: the new approach to recertification in pediatrics. Pediatric Diplomates. 2001;Fall:1-4
Carol Carraccio, MD *; Robert Englander, MD, MPH ([double dagger]); Susan Wolfsthal, MD *; Christine Martin, PhD *; and Kevin Ferentz, MD *
From the * University of Maryland School of Medicine, Baltimore, Maryland; ([double dagger]) University of Connecticut School of Medicine, Hartford, Connecticut.
Received for publication Apr 14, 2003; accepted Sep 10, 2003.
Address correspondence to Carol Carraccio, MD, Department of Pediatrics, University of Maryland, 22 South Greene St, Baltimore, MD 21201. E-mail: ccarraccio@peds.umaryland.edu
PEDIATRICS (ISSN 0031 4005). Copyright [c] 2004 by the American Academy of Pediatrics
ABBREVIATIONS. ACGME, Accreditation Council for Graduate Medical Education; ABMS, American Board of Medical Specialties; APPD, Association of Pediatric Program Directors; ABP, American Board of Pediatrics; PGY, postgraduate year.
COPYRIGHT 2004 American Academy of Pediatrics
COPYRIGHT 2004 Gale Group
