Refuting the False Trope on Clinical Courses and Bar Passage

Posted on January 8, 2020 by rkuehnblog

Robert Kuehn, Washington University School of Law

It has been observed that “the fewer the facts, the stronger the opinion.” Until recently, this could be said about the possible influence of enrollment in clinical courses on a student’s likelihood of passing the bar examination. While there was a shortage of empirical studies on any possible relationship, there have been plenty of opinions on how taking those courses might be harmful, opinions often reflected in graduation restrictions on clinical courses and requirements for bar subject-matter courses.

But, there are now significantly more facts to refute those opinions. Two recent, large-scale studies have both found no relationship between the number of law clinic or externship courses or credits a law graduate took and her likelihood of passing the bar exam.

In a forthcoming article in the Journal of Legal Education, academic records of ten years of law school graduates of Washington University in St. Louis and Wayne State University were reviewed for any relationship between the number of law clinic, externship, or, more generally, experiential courses or credits and bar passage. After first accounting for the possible influence of law school grades on bar passage (the most significant predictor of bar success), the study found no correlation at either school between law clinic or externship enrollment and bar passage — no relationship between participation in a law clinic or externship and passage, none between the number of clinical courses and passage, none between the number of clinical credits and passage, and no evidence that students graduating with lower GPAs were disproportionately enrolling in those courses as a way to avoid doctrinal courses (another not uncommon trope). This lack of any relationship was in spite of increased enrollment in experiential courses at both schools over the ten-year period and decreased enrollment in courses teaching material tested on the bar (referred to as bar subject-matter courses).

The article notes that nationwide data on experiential course enrollment and bar passage also belie any claim the two are related. That data indicate that as enrollment in experiential courses was increasing from 2006-2016, bar passage percentages were fairly steady and that the recent decline in passage coincided with decreased, not increased, enrollment in those courses.

A recent study commissioned by the California State Bar found a similar lack of relationship between law clinic and externship courses and bar exam performance. The study reviewed law school coursework and performance on three July exams for over 7,500 bar applicants from eleven California schools. It found no relationship between the number of academic credits from law clinic courses and exam performance, either across all schools or even when reviewing schools separately. Similarly, there was no relationship between the number of externship or internship credits and performance, again when examined across all schools or within schools. The broad range of entering credentials at the eleven schools, and lack of a relationship even within those schools, indicates that the results should be applicable to most law schools, including those with lower LSATs and undergraduate GPAs for entering students.

The study results from Washington University/Wayne State and the California State Bar are similar to smaller studies at Texas Tech and the University of Denver that also reported no statistically significant relationship between enrollment in a law clinic or externship course and bar passage.

The Washington University/Wayne State and California State Bar studies further revealed that opinions about the value of bar subject-matter courses should be moderated. There were small correlations at both schools between the number of bar subject courses and bar passage. But this result (explaining less than 5% of the variability in bar outcomes) was only for low performing students and additional courses showed no marginal benefit once students took the school’s average number of bar courses.

The California State Bar study focused on whether taking a specific course was related to performance on the bar exam topic taught in those courses. It found that neither attendance nor performance in courses covering any of the 13 bar-related topics was related to performance on the corresponding California bar exam or Multistate Bar Exam content covering that subject.

Studies at other schools also indicate that enrollment in bar subject-related courses do not support broad claims about the benefit of taking those courses.

It is time to put away the misinformed trope of participation in law clinic and externship courses harming a student’s chances of passing the bar exam and let the facts do the talking. Law schools should recognize and students should be told they can obtain valuable preparation for the practice of law by enrolling in clinical courses without affecting their likelihood of passing the bar exam.

Filed under: Uncategorized | Tagged: , , , , , , , , , , , , , | Comments Off on Refuting the False Trope on Clinical Courses and Bar Passage

An Overview of “A Study of the Relationship Between Law School Coursework and Bar Exam Outcomes”

Posted on September 26, 2019 by Jessica N. Haller

Robert Kuehn, Professor of Law at Washington University School of Law in St. Louis, and David Moss, Associate Clinical Professor at Wayne State University Law School, recently conducted a large-scale study looking at the relationship between clinical/experiential or bar subject-matter courses and bar passage success in a paper entitled, “A Study of the Relationship Between Law School Coursework and Bar Exam Outcomes.” As a law student currently enrolled in a clinic, this study immediately piqued my interest.

This study was in response to fear that bar passage rates were down because of rising enrollment in “experiential courses” as opposed to “bar-subject courses.” Law schools began pushing students to enroll in more of these bar-subject courses to correct this so-called issue. However, Professor Kuehn and Professor Moss observed that there was no evidence to suggest that taking more bar-subject courses was appropriate advice for all students. Their study looked at this missing evidence between bar-subject courses and experiential learning and bar exam outcomes for ten years between two law schools: Washington University School of Law (WashU) and Wayne State University Law School (Wayne State). Both schools only require the designed first year courses and the upper-level writing courses mandated by ABA accreditation standards.

Previous studies performed in states like Texas, Colorado, and California looked at the effects of coursework and bar passage rates. These studies did not support the claim that taking more bar-tested law school course improve chances of passing on the first attempt. Notably, a study done in Indiana concluded, “simply forcing lower-performing students to take more upper division bar-subject courses will not solve the bar examination failure problem.”

The first goal of the present study was to determine whether a graduate’s enrollment in elective experiential courses was related to first-time bar passage success. Next, it was to assess whether enrollment in elective courses that cover bar subjects was related to bar success.

Data was collected from law school graduates from 2006-2015. The following table outlines the number of graduates with LSAT scores and bar passage rates between the two schools:

The next table looked at graduate characteristics such as undergraduate GPA, LSAT score, 1L GPA, and law GPA and their correlation with bar passage:

It wasn’t until 2005 that the ABA began requiring graduates to receive professional skills instruction with as little as one credit satisfying the requirement. In 2014, the ABA changed this to require six credits beginning with 2019 graduates. The study authors decided to track enrollment in skills courses versus bar passage over this time period.

The table above reveals a solid line depicting that average bar passage percentages were steady from 2006-2013 (this is when experiential course enrollment increased by over 50%). During the significant rise in experiential enrollment, bar passage percentages were largely steady. “Therefore, efforts to link declining nationwide bar passage rates to the rise in experiential course enrollment are not supported by national statistics.” A more likely contributing cause for bar passage declines since 2014 is weaker credentials of incoming 1Ls.

At WashU, it was found that while taking at least the average number of bar courses is associated with increased likelihood of passing the bar, there was no statistically significant increase in bar passage associated with bottom-quartile LGPA graduates who took more than the school’s average. This was similar with graduates in the bottom half of their class at Wayne State. Results for both schools indicate that graduates in bottom quartile who take fewer than the average number of bar courses at their school were associated with a significant increase in bar failure. Further, at both schools, students entering with scores lower than 150 were associated with pass rates significantly below the school’s average.

This study concluded that the claim that the dramatic decline in bar passage rates is due to law students taking more experiential courses or fewer bar-related courses is not supported. It characterized efforts to cap experiential credits in order to improve bar passages rates are “misguided,” warning that schools should not expect that “mere exposure” to more bar courses will significantly improve bar passage rates.

Also see “Legal Skills Prof Blog” and “TaxProf Blog” for more posts on this study

Filed under: Uncategorized | Tagged: , , , , , , , , , , , | 2 Comments »

Fostering Student Success: Part II -Possible Actionable Steps to Encourage Growth Mindsets

Posted on September 5, 2019 by Jessica N. Haller

The opinions expressed within this article are the personal opinions of the author.

By Sara J. Berman, Director of Academic and Bar Success Programs at the AccessLex Institute’s Center for Legal Education Excellence; SSRN author page https://ssrn.com/author=2846291

As was detailed in Part I of this post on Fostering Student Success, we must meaningfully reward those who do the hard work and actually end up achieving the requisite skills and doctrine mastery at some point (any point!) before graduation. Those who take a bit longer to catch on must be given true opportunities to improve so that they see struggling to learn as evidence of powerful grit and a stepping stone to a lifetime as a successful professional, rather than a predictor of future failure.  Below are a number of possible actionable steps we might consider piloting and studying.

First, we might encourage growth mindsets by listing grades as AGP (annual grade points) rather than cumulative GPA (grade point average). Every year would provide a new, level playing field for students, and, employers would readily see whose grades increased, and by how much each year. (Note: Scholarship comprehensively critiquing grading and class ranking systems dates back some time. [1] The suggestions here simply point to “low-hanging fruit” interventions.) A natural criticism of this approach is that first-year courses tend to be required and are thus an apples-to-apples comparison, while upper-division courses vary widely and often have looser grading policies. Too many 2L students who see Cs turn to Bs falsely attribute this “improvement” to their own effort when grade increases actually stem from “easier” courses and/or more lenient grading.  Nonetheless, there could be a great psychological benefit to having a “clean slate” each year, with new opportunities in 2L and 3L to be at the top of the class. Prestigious and financially generous awards could be given to students whose GPAs have increased the most from the first year to the third year. And, employers could still see grades in particular courses and full transcripts as desired.

Second, we could study the effect of eliminating class ranking altogether. Justified, as is GPA, by the “needs” of employers, class ranking also fosters a fixed mindset, competitively boxing students into “winners” (those at the top of the class) and “losers” –those at the bottom who  may internalize defeat and, far too often, treat low ranking as a predictor of bar exam failure (which in turn may become a self-fulfilling prophecy).[2] Are class rankings necessary? What pedagogical purpose do they serve? Some medical schools are moving to a pass/fail model[3] with less emphasis on relative rank.[4] This appears to be reducing some of the stress associated with mental health challenges in these similarly high-pressured graduate programs[5] without affecting academic performance or accomplishment.[6] Some (mostly elite) law schools do not rank students. Should others experiment as well?  The main advantage appears to be providing a triage system for potential employers, (e.g. “We only hire from the top 25% of the class.”). Yet recent studies[7] show that what many legal employers want in new lawyers includes so-called “soft skills,” not measured by grades or class rankings.[8]  If this is the case, might we better serve employers’ needs by creating rubrics to measure professionalism and practical lawyering skills?  Highlighting how much a student’s grades have improved from 1L to graduation could help employers measure resiliency, while actually encouraging improvement by stemming some of the “why bother” mentality of those who turn off after receiving low 1L grades.  

Third, let us endorse studies that pilot tests of non-cognitive skills, such as those LSAC is undertaking and those inspired by the Shultz and Zedeck studies.[9] And let us support and laud efforts to showcase (in part for potential employers) the wide range of student skills on display in lawyering competitions.[10]

Fourth, let us identify and study other creative ways to assist employers while breaking vicious, defeatism cycles that thrive in our current system. I have long encouraged graduating classes with the aspirational challenge of 100% bar passage, reminding them that while class ranking forced some to the top and others to the bottom, every graduate can pass the bar exam first time around.  (Recall the old joke: “Question: What do you call the person who was last in his class in medical school? Answer: Doctor!”).  I also urge law graduates to help each other –with a “rising tide lifts all boats” philosophy and with the learning science-backed truth that teaching another is often the best way to learn.

Fifth, we might pilot the administration of comprehensive exams at the end of each year of law school.  These would encourage students to review and be re-tested on key subjects, “building mental muscle” over time so that they learn to master materials they may only have understood superficially when first exposed.  Awards could be given to every student who achieved high scores on these “comps,” rewarding those who caught on later as well as those who caught on initially.

Sixth, we could develop a national pre-bar exam (what I call the “NPBE”), similar to the PSAT, which would allow 2L law students a high-stakes “practice exam” which schools could use as a diagnostic and formative assessment so that law graduates do not have to fail the bar exam in order to realize how much improvement they really need to pass, in skills, substance, time management, mindset, and more.[11] Like the PSAT with its National Merit Scholar incentives, the NPBE could award scholarships to those with low 1L grades who overcome challenges and perform exceptionally well on the NPBE.

Perfect pass rates are not impossible on the law school side (though I understand limitations that may result from certain jurisdictions’ cut scores), especially when considering cumulative rather than first-time bar passage, per the new ABA Standard 316.[12]  But widespread student success requires more than mouthing “grit” and “persistence” mantras.  We must actively foster institution-wide expression of and action supporting the belief that every student who is not academically dismissed can pass the bar exam.  We must equip all students who graduate from ABA law schools to pass the bar first time around.  And, if we truly hope to so equip our law students, their self-perceptions simply may not be allowed to become fixed after first semester grades. 

[1] Barbara Glesner Fines, Competition and the Curve, 65 UMKC L. Rev 879 (1997); Jay M. Feinman, Law School Grading, 65 UMKC L. Rev. 647, 656 (1997); Jerry R. Foxhoven, Beyond Grading: Assessing Student Readiness to Practice Law, 16 Clinical L. Rev. 335 (2009); Heather D. Baum, Inward Bound: An Exploration of Character Development in Law School, 39 UALR L. Rev. 25 (2016).

[2] Query whether research presented at AALS (January 2018) by Professor Robert R. Kuehn (Washington University in St. Louis) suggests this, given results of students with identical entering LSAT scores failing the bar where they were at the bottom of the class and passing where they were at the top of the class.

[3] Casey B. White and Joseph C. Fantone, Pass–fail Grading: Laying the Foundation for Self-Regulated Learning, 15 Advances in Health Sci. Educ. 469 (2010).

[4] John P. Bent et al., Otolaryngology Resident Selection: Do Rank Lists Matter? 144 Otolaryngology-Head & Neck Surgery 537 (2011).

[5] Daniel E. Rohe et al., The Benefits of Pass-Fail Grading on Stress, Mood, and Group Cohesion in Medical Students, 81 Mayo Clinic Proc. 1443 (2006); see also Robert A. Bloodgood et al., A Change to Pass/Fail Grading in the First Two Years at One Medical School Results in Improved Psychological Well-Being, 84 Acad. Med. 655 (2009); Francis Deng and Austin Wesevich, Pass-fail is here to stay in medical schools. And that’s a good thing, KevinMD.com (Aug. 3, 2016).

[6] B. Ange et al., Differences in Medical Students’ Academic Performance between a Pass/Fail and Tiered Grading System, 111 S. Med. J. 683 (2018).

[7] Alli Gerkman & Logan Cornett, Foundations for Practice: The Whole Lawyer and the Character Quotient, AccessLex Inst. Res. Paper Series No. 16-04 (2016).

[8] Bryant G. Garth, Notes on the Future of the Legal Profession in the United States: The Key Roles of Corporate Law Firms and Urban Law Schools, 65 Buff L. Rev. 287 (2017).

[9] Marjorie M. Shultz & Sheldon Zedeck, Predicting Lawyer Effectiveness: A New Assessment for Use in Law School Admission Decisions, CELS 2009 4th Ann. Conf. on Empirical Legal Stud. Paper (2009).

[10] Sherry Y. English, Cincinnati Law hosts nation’s first, only law student case competition, UC News (Jan. 10, 2019),https://www.uc.edu/news/articles/2019/01/n2059715.html.

[11]As I often say, would anyone mount a Broadway show without a dress rehearsal? Do athletes compete in the Olympics without high-profile pre-competition practice?  No!  Yet we wait until after law school and generally outsource to bar reviews the only sort of organized practice runs for the highest stakes law exam of all.

[12] Two Indiana law schools soar on ultimate bar passage rate, Ind. Law. (April 22, 2019),https://www.theindianalawyer.com/articles/50047-two-indiana-law-schools-soar-on-ultimate-bar-passage-rate.

Filed under: Uncategorized | Tagged: , , , , , , , , , | 2 Comments »

%d bloggers like this: