It’s not unusual for a provost or a colleague or a relative at Thanksgiving to ask a legal academic why law students have so much trouble passing the bar exam when the pass rates for medical students are usually in the high 90th percent. The short answer to that question is that the two processes are completely different—and there’s no obvious trick, technique, or intervention that could convert our bar passage rates into their licensure passage rates. For one thing, it’s the wrong question. “Passing” the medical licensing exams is certainly important, but unlike the “all or nothing” process of passing the bar exam, the score achieved on Step 1 affects medical students’ entire career path. But there is a lot to learn about the methods that medical schools use in studying the very few students who have trouble as well as how they evaluate the effect of changes to their curriculums on scores on the licensing exams.
Quick recap on professional licensing—future doctors take a series of three exams over the first six years of their undergraduate medical education and the start of their residency. (more links in a post I wrote earlier this year here). The exams are almost entirely national although the actual process of being licensed is conducted on a state by state basis. Law students take a licensing exam in the state where they intend to practice upon graduation. For purposes of this post, the closest analogy to the bar exam is the more academic Step One students take during their second year of medical school. Like our NCBE, the National Board of Medical Examiners which produces United States Medical Licensing Examination works with medical licensing boards and depends on their confidence. It issues annual reports.
The focus of this post is on the methods that medical schools use to study the small number of their students who do have trouble passing the licensing the exams as well as the factors that can affect the scores students achieve. I’ve tried to focus on articles outside of paywalls, and would certainly encourage you to conduct your own searches in the various data bases to which you have access. There are several journals devoted directly to studying medical education—although these articles can pop up anywhere.
Medical educators use a wide range of research techniques to learn more about students who struggle with licensure exams. Like us, medical schools would prefer students pass the first time and many articles like this one look for characteristics who fail the first time but eventually pass. Others look for characteristics of students at risk for failure here and here or even what students think of the exam. Another area for inquiry involves the role stress plays in the score students achieve. In partnership with social scientists at our schools or in our communities, we too could be conducting studies to help us learn more about students who face difficulty passing the bar exam. These studies can be part of graduate student work or may even be funded by groups like Access which is making money available to study bar passage.
The actual reason the medical school pass rates are so high, though, may not be all that helpful.
It’s not just because they are able to limit admission to students who have already demonstrated an ability to score very highly on the MCAT. A test that is much more similar to step 1 than the bar exam is to the LSAT. Indeed, medical schools have direct input in both the MCAT and the Licensing Exams—so when one changes, the other can too. And it’s not clear that anything in the curriculum makes a difference at all—the industry offering study aids and licensure prep courses dwarfs the bar prep and study aid market to a point where students often start studying for the licensing exams before the first day of medical school.
But if it is the curriculum, it’s important to remember the vast difference in time scale between medical and legal education. We have students for three years post B.A. Medical schools in the U.S. plan their curriculum based on 8 plus years of increasingly specialized medical education. They are therefore comfortable holding off on the direct teaching of practice skills for the first two years while they are aligning their curriculum with the content of the Step 1 exam.
Even Step 1, though, is far more focused on practice than on knowledge accumulation or deliberately confusing question formulations that characterize the bar exam. Step 2, the second round of licensing exams prior to graduation medical school, go past paper and pencil in that they actually test students’ ability to conduct exams and exercise medical judgement. Another reason for the high pass rate is that most medical schools have stopped developing their own tests and instead use assessment instruments (shelf exams) provided by the same company that produces the exam. Sure, there is grumbling and criticism about content & timing of the licensing exams, but medical schools work hard to make sure that their curriculums are aligned with the content of the exams. Finally, medical education is extremely self-reflecting–they are constantly aware of the risks that come from confusing correlation and causation. How do you know that a change in one part of the curriculum is the cause of a change in test scores? You run Pearson correlations followed by stepwise linear regressions. Seeing is not believing when comes to identifying factors that affect performance on licensure exams. Look here, here, here, and here for studies evaluating curriculum changes. They take nothing for granted—does attendance make a difference, does flipping classrooms really work? Does reducing the number of hours spend in the anatomy lab reduce USMLE scores?
Another standard practice in medical schools is curriculum mapping— an essential first step for any school that wants to understand what they are teaching—let alone make changes. Like all maps, curriculum maps are DESCRIPTIVE, not PROSCRIPTIVE. Here is Harvard’s curriculum map, but you can find examples on the home page of just about every U.S. Medical School.This is a an article walking through how to map a curriculum.
So what’s helpful to us isn’t so much what medical schools are doing, but how they are evaluating themselves.
In recap, neither I nor anyone else who has ever practiced law thinks it would be a good idea to emulate medical schools by fully aligning our curriculum with the bar exam so as to turn the three years of law school into one extended bar prep course. Among other reasons, the material tested on the bar is quite static and doesn’t reflect the realities of today’s law practice. It also wouldn’t make much sense for schools whose students take the bar exam in many different jurisdictions. Also, the bar exam is just not equivalent to the three rounds of USMLE exams in actually testing both the knowledge and application of knowledge needed to be a successful lawyer. If it was, we wouldn’t hear so many complaints about how students who have passed bar are never-the-less not “practice ready.”
Tomorrow—where can we get the help we need to find out this information, and who is going to pay for it? Spoiler--Access Lex has a program.
Filed under: Bar Exam, Best Practices, Outcomes & Assessment Techniques, First Year Law School Innovations, Outcomes & Assessment Techniques | Tagged: bar exam, Medical School, Regression Analysis, USLME | 7 Comments »
Best Practices for Legal Education 