Can Physicians Self-Assess? Not Very Well.
This was the conclusion reached by some 200 participants in a CME Summit, sponsored earlier this month in Chicago by the U.S. Council of Medical Specialty Societies. They had no choice but to agree with the reports of two Canadian researchers, based on meta-analyses.
David A. Davis MD of the University of Toronto, recognized as the dominant research expert in continuing medical education, summarized the findings he and his colleagues reported in a recent article in the Journal of the American Medical Association:
Kevin Eva PhD, of the program for Educational Research and Development at McMaster University, echoed these findings, based on an article that he and an associate wrote in Academic Medicine (October 2005 Supplement):
Eva then listed some specific implications for physicians based on a search of related literature in self-efficacy and self-concept, cognitive theory, social cognition, expert performance and reflective practice. Here are some of his findings:
Once attendees at the conference recovered from the shock of discovering that self-assessment in itself is not a desirable goal, they focused on two cardinal rules for directed self- assessment:
Robert Galbraith MD, director of the Center for Innovation of the National Board of Medical Examiners (U.S.), offered his approach to these principles, based on development of practice/work profiles for individual physicians, linking these to multiple-choice questions to test knowledge, and in turn adding external assessment in the forms of comparison to practice norms and mentoring. These steps, he suggested, can assist the physician in closing the gaps in knowledge and skills in a process of continuous improvement. And continuous improvement requires continuous monitoring of performance.
There are parallels here to the new system of revalidation in the UK, though this was not mentioned at the summit. Whether the UK system is working or not is another question. Here is a recent e-mail query to members of the TUFH (Towards Unity for Health) Network: "We are looking for any published evidence on the impact of CPD/revalidation on the performance of (health) professionals."
On a macro basis, there are many knowledge self- assessment exams available, primarily through specialty organizations such as the American College of Physicians, American Society of Clinical Oncology, American Psychiatric Association, and others. These are based on the most up-to-date evidence- based guidelines. But the problem is linking these to an individual's practice profile, which is not readily available in most countries.
The American Board of Medical Specialties is struggling with this problem as it pursues a goal of maintenance of certification based on self-evaluation every 1-3 years and an exam only every 10 years. One element is a practice improvement module based on chart review, a patient survey and a practice survey. Some specialty societies have begun to implement this on an experimental basis.
Then there is the question of what governments will require. As noted, the UK has imposed a mandatory revalidation system combining self-assessment with practice profiling, aided by a mentor. Canada is moving in a similar direction. The U.S. Federation of State Medical Boards is seeking to move from a mandatory credit-hour basis in most states to measurement of the ongoing competence of a physician -- one that the medical community, legislators and voters can accept. But many countries still have no relicensure requirements in place.
And we haven't even mentioned the role of hospitals in requiring evidence of competence and monitoring performance.
The lesson from the CME Summit is clear: We can no longer rely on the individual physician to structure his/her own approach to CME based on self-assessment. There needs to be another component, however crudely designed, to provide some outside direction to identifying individual needs based on a practice profile.