Latest News

Can Physicians Self-Assess? Not Very Well.
WentzMiller & Associates Global CME Newsletter
11/27/2006

This was the conclusion reached by some 200 participants in a CME Summit, sponsored earlier this month in Chicago by the U.S. Council of Medical Specialty Societies. They had no choice but to agree with the reports of two Canadian researchers, based on meta-analyses.

David A. Davis MD of the University of Toronto, recognized as the dominant research expert in continuing medical education, summarized the findings he and his colleagues reported in a recent article in the Journal of the American Medical Association:

• CME credit, relicensure (revalidation) and recertification are linked to the abilities of physician to assess their own needs and select learning activities to meet those needs.
• In a literature search of comparisons between self and external assessment, 13 of 20 studies demonstrated little, no or an inverse relationship; 7 a positive association.
• A number of studies found the worst accuracy in self-assessment among those who were the least skilled.
• Conclusion: The preponderance of evidence suggests that physicians have a limited ability to accurately self-assess.

Kevin Eva PhD, of the program for Educational Research and Development at McMaster University, echoed these findings, based on an article that he and an associate wrote in Academic Medicine (October 2005 Supplement):

• The archetype of the self-regulating professional will reflect regularly on practice, self-assess gaps, seek to redress these, and incorporate new knowledge and skills in practice.
• All but the very highest performers tend to overestimate their ability.
• Those most in need of improvement are those least likely to know.
• Conclusion: A critical premise (self-assessment) underlying the concept of self-regulation is unsupportable.

Eva then listed some specific implications for physicians based on a search of related literature in self-efficacy and self-concept, cognitive theory, social cognition, expert performance and reflective practice. Here are some of his findings:

• We can never create a good "self-assessor" -- and probably shouldn't try.
• Learners rarely spontaneously face the failures that enable better judgements of learning.
• We know little about the extent to which physicians intentionally seek external assessments.
• Doctors need coaches to motivate them and to help them understand the boundaries of their abilities.
• Reflection in practice is highly important for ensuring safe and effective practice.

Once attendees at the conference recovered from the shock of discovering that self-assessment in itself is not a desirable goal, they focused on two cardinal rules for directed self- assessment:

1. Determine how to combine self and external assessment into a working system that truly will improve physician competence.
2. Convince physicians and educators that external assessments are legitimate and should be sought.

Robert Galbraith MD, director of the Center for Innovation of the National Board of Medical Examiners (U.S.), offered his approach to these principles, based on development of practice/work profiles for individual physicians, linking these to multiple-choice questions to test knowledge, and in turn adding external assessment in the forms of comparison to practice norms and mentoring. These steps, he suggested, can assist the physician in closing the gaps in knowledge and skills in a process of continuous improvement. And continuous improvement requires continuous monitoring of performance.

There are parallels here to the new system of revalidation in the UK, though this was not mentioned at the summit. Whether the UK system is working or not is another question. Here is a recent e-mail query to members of the TUFH (Towards Unity for Health) Network: "We are looking for any published evidence on the impact of CPD/revalidation on the performance of (health) professionals."

On a macro basis, there are many knowledge self- assessment exams available, primarily through specialty organizations such as the American College of Physicians, American Society of Clinical Oncology, American Psychiatric Association, and others. These are based on the most up-to-date evidence- based guidelines. But the problem is linking these to an individual's practice profile, which is not readily available in most countries.

The American Board of Medical Specialties is struggling with this problem as it pursues a goal of maintenance of certification based on self-evaluation every 1-3 years and an exam only every 10 years. One element is a practice improvement module based on chart review, a patient survey and a practice survey. Some specialty societies have begun to implement this on an experimental basis.

Then there is the question of what governments will require. As noted, the UK has imposed a mandatory revalidation system combining self-assessment with practice profiling, aided by a mentor. Canada is moving in a similar direction. The U.S. Federation of State Medical Boards is seeking to move from a mandatory credit-hour basis in most states to measurement of the ongoing competence of a physician -- one that the medical community, legislators and voters can accept. But many countries still have no relicensure requirements in place.

And we haven't even mentioned the role of hospitals in requiring evidence of competence and monitoring performance.

The lesson from the CME Summit is clear: We can no longer rely on the individual physician to structure his/her own approach to CME based on self-assessment. There needs to be another component, however crudely designed, to provide some outside direction to identifying individual needs based on a practice profile.

Back to Latest News page