Best Practice Contributors Highlighted in Best Articles of 2019

A big congratulations to our very own bloggers, Jennifer Bard and Benjamin Madison, for being featured on the TaxProf Blog!

Jennifer Bard’s article, “Are the Students Failing the Bar Exam Today Canaries in the Coal Mine warning us of a More General Need to Change Legal Education?” and Benjamin Madison’s article, “New Rubrics Available to Help Law Schools that Have Adopted Learning Outcomes Related to Professional Identity Formation” were both listed as TaxProf Blog’s “Best Legal Education Articles of 2019.”

Refuting the False Trope on Clinical Courses and Bar Passage

Robert Kuehn, Washington University School of Law

It has been observed that “the fewer the facts, the stronger the opinion.” Until recently, this could be said about the possible influence of enrollment in clinical courses on a student’s likelihood of passing the bar examination. While there was a shortage of empirical studies on any possible relationship, there have been plenty of opinions on how taking those courses might be harmful, opinions often reflected in graduation restrictions on clinical courses and requirements for bar subject-matter courses.

But, there are now significantly more facts to refute those opinions. Two recent, large-scale studies have both found no relationship between the number of law clinic or externship courses or credits a law graduate took and her likelihood of passing the bar exam.

In a forthcoming article in the Journal of Legal Education, academic records of ten years of law school graduates of Washington University in St. Louis and Wayne State University were reviewed for any relationship between the number of law clinic, externship, or, more generally, experiential courses or credits and bar passage. After first accounting for the possible influence of law school grades on bar passage (the most significant predictor of bar success), the study found no correlation at either school between law clinic or externship enrollment and bar passage — no relationship between participation in a law clinic or externship and passage, none between the number of clinical courses and passage, none between the number of clinical credits and passage, and no evidence that students graduating with lower GPAs were disproportionately enrolling in those courses as a way to avoid doctrinal courses (another not uncommon trope). This lack of any relationship was in spite of increased enrollment in experiential courses at both schools over the ten-year period and decreased enrollment in courses teaching material tested on the bar (referred to as bar subject-matter courses).

The article notes that nationwide data on experiential course enrollment and bar passage also belie any claim the two are related. That data indicate that as enrollment in experiential courses was increasing from 2006-2016, bar passage percentages were fairly steady and that the recent decline in passage coincided with decreased, not increased, enrollment in those courses.

A recent study commissioned by the California State Bar found a similar lack of relationship between law clinic and externship courses and bar exam performance. The study reviewed law school coursework and performance on three July exams for over 7,500 bar applicants from eleven California schools. It found no relationship between the number of academic credits from law clinic courses and exam performance, either across all schools or even when reviewing schools separately. Similarly, there was no relationship between the number of externship or internship credits and performance, again when examined across all schools or within schools. The broad range of entering credentials at the eleven schools, and lack of a relationship even within those schools, indicates that the results should be applicable to most law schools, including those with lower LSATs and undergraduate GPAs for entering students.

The study results from Washington University/Wayne State and the California State Bar are similar to smaller studies at Texas Tech and the University of Denver that also reported no statistically significant relationship between enrollment in a law clinic or externship course and bar passage.

The Washington University/Wayne State and California State Bar studies further revealed that opinions about the value of bar subject-matter courses should be moderated. There were small correlations at both schools between the number of bar subject courses and bar passage. But this result (explaining less than 5% of the variability in bar outcomes) was only for low performing students and additional courses showed no marginal benefit once students took the school’s average number of bar courses.

The California State Bar study focused on whether taking a specific course was related to performance on the bar exam topic taught in those courses. It found that neither attendance nor performance in courses covering any of the 13 bar-related topics was related to performance on the corresponding California bar exam or Multistate Bar Exam content covering that subject.

Studies at other schools also indicate that enrollment in bar subject-related courses do not support broad claims about the benefit of taking those courses.

It is time to put away the misinformed trope of participation in law clinic and externship courses harming a student’s chances of passing the bar exam and let the facts do the talking. Law schools should recognize and students should be told they can obtain valuable preparation for the practice of law by enrolling in clinical courses without affecting their likelihood of passing the bar exam.

Active Retrieval Practice: Known to Improve Learning but Underappreciated?

Exam time has arrived in law schools.  Students who want to excel on exams (and later, as attorneys) would do well to try out active retrieval practice.  To understand the value of retrieval practice, some brief discussion of well-established cognitive science is necessary.  Learning involves (1) taking knowledge into short-term working memory, and then (2) moving it from working memory to long-term memory by actively using the knowledge.[1]  In their excellent book, Teaching Law by Design, Dean Michael Hunter Schwartz, Professor Sophie Sparrow, and Professor Gerald Hess explain this process of storing learning in cognitive schema.[2]  They liken schemata to a “folder system[] provided for users of computer operating systems.”[3]  As they observe, however, storing knowledge “isn’t enough.  To analyze a problem, students must recall (“retrieve”) what they have learned and use that learning . . . .”[4] 

Research on cognition demonstrates that meaningful learning in any discipline requires the learner to perform some form of active retrieval exercises to be able to use the knowledge in analyzing and solving problems.   Active retrieval methods are ways in which the learner recalls knowledge and uses the recalled knowledge to solve problems or answer questions.[5]  Recalling for mere “knowledge checks,” sometimes called rote learning, is not effective.[6]   In the law school arena, a student can recite a memorized rule but not be able to apply it to fact patterns in a way that shows understanding.   Effective retrieval-based learning activities require the student to solve problems or to answer questions.  By doing so, the learner strengthens her understanding  of, and ability to recall, the knowledge.[7]  In law school, mid-term exams require students to recall information at least in mid-semester.  The problem there is that neuroscience shows a marked forgetting curve: if learning is not retrieved within a few days of its being stored, the knowledge is lost and must be relearned.[8]  Indeed, retrieving and using the knowledge are the critical parts of developing meaningful learning. 

               Spurred by the adoption of ABA Standard 314, my colleagues and I have been giving mid-term exams and using a variety of interim assessments designed to have students to recall information from previous classes. I regularly have a mid-term that includes multiple-choice, essays, or both.  The exams are graded and students receive a model answer. I discuss with students the answers to the assessments and common mistakes (e.g., failing to state rules accurately, insufficient application of facts in supporting one’s analysis).   The exam and follow-up discussions achieve the goal of providing the “meaningful feedback” that ABA Standard 314 seeks.   The mid-term is also summative. My experience is that many students do not take a practice, ungraded mid-term seriously. Having the exam count, but not so much as to prevent a student from recovering from a poor exam, helps to ensure that students prepare for and spend time on the mid-term.

               Another way that I have incorporated active recall practice is by using multiple-choice polling questions.  In the first class after we finish a course segment, we begin the class with multiple-choice questions that students answer by polling.    I ensure participation by recording the polling, by student, and including their responses (or lack thereof) as a class-participation part of the grade.  In answering the questions through polling, students must recall knowledge to analyze the question and reach a conclusion.   For example, after completing the study of removal jurisdiction and procedures, I use a series of multiple-choice polling questions that explore the many nuances of removal.   These sessions provide “meaningful feedback” to both students and to me.   If everyone misses a question, you can be sure I go back to discuss the area.  I also encourage anyone who missed a question in these polling sessions to meet with me after reviewing the topic addressed by the question.

               I urge students not to rely solely on the mid-term and the polling sessions as a means of ensuring they have learned material well.  Instead, I highly recommend preparing answers to essays under timed conditions.  At times I provide a model answer after they practice an essay. I also invite students to meet with me to go over their answers.   In these discussions, I almost always find some area in which a student has a mistaken understanding of a rule or concept.  If we did not uncover that misunderstanding, a student could repeatedly recall a flawed rule or approach.  Hence, I appreciate more than ever the wisdom of Standard 314’s emphasis on formative assessment.   After resolving any misunderstandings, I encourage a student to rewrite an answer.  That allows the student to revisit the topic and solidify her understanding.  Indeed, the act of writing itself helps students to embed the rules and concepts more firmly in their memory.[9] 

               What is true for law school is also true for the Bar Exam.   Last summer Sara Berman and I created a podcast for the ABA on practicing tests (essays, Multistate Performance Tests [MPTs], etc.) as some of the most effective ways to prepare for a state’s Bar Exam.[10]  Ideally, a student learns everything she needs to know in law school Bar review is just that–review.  More often, Bar applicants have a vague recollection of rules and concepts from their time in law school.  In other words, Bar applicants often find themselves relearning rules and concepts.  In so doing, they will learn more effectively by practicing an essay or MPT answer and by submitting these answers for grading to their Bar Preparation company and the faculty at their law school who help Bar applicants.  In doing these exercises, they benefit for at least two reasons.  First, it will identify areas in which the applicant’s knowledge of rules and concepts is so weak that she cannot answer a question.   Knowing that, the applicant can review that area and know that she needs to do so.  Second, exercises such as practice essay answers will require applicants to recall the rules they do know and apply them.  The more they do so, the more likely they are to remember them on the Bar. 

               Repeated, active retrieval practice is one of the best ways to learn to perform on exams or in law practice.  Yet, despite the data showing its effectiveness, such practices are not the norm in higher education.[11]  The practice is likely not the norm in law schools.   Standard 314 ought to help to some extent increase active retrieval before the end of the law school semester.  Yet, at present students are not spending their time as wisely as they could. Instead of preparing detailed outlines, and memorizing rules or flash cards, they would learn more from methods that require them to recall and apply legal rules and analysis.   Indeed, one might say this fact is one of the best kept secrets in law school. Perhaps it is time to let this secret out.


[1] Michael Hunter Schwartz, Sophie Sparrow & Gerald Hess, Teaching Law by Design 4­–7 (Carolina Academic Press 2009).

[2] Id.

[3] Id. at 5–6.

[4] Id. at 6.

[5] Jeffrey D. Karpicke, A Powerful Way to Improve Learning and Memory: Practicing Retrieval Enhances Long-Term, Meaningful Learning, American Psychological Association (2016), available at https://www.apa.org/science/about/psa/2016/06/learning-memory (last checked Dec. 6, 2019).

[6] Id.

[7] Id.

[8] Louis N. Schulze, Jr., Using Science to Build Better Learners: One School’s Successful Efforts to Raise its Bar Passage Rates in an Era of Decline, 68 J. Legal Educ. 230, 245 (Winter 2019).

[9] Bryan Goodwyn, The Magic of Writing Stuff Down, 75 Educational Leadership 78–79 (April 2019).

[10] Sara Berman and Ben Madison, Practice Makes Passing, Episode 6 of American Bar Associations Path to Law Student Well-Being Podcast series, available at  https://www.spreaker.com/show/path-to-law-student-well-being  (June 22, 2019).

[11] According to Dr. Karpicke, college students likewise use rote learning methods more than they use active retrieval exercises.  See Karpicke, supra note 5.

Guiding Students From Law School Into the World

It seems that one of the things we law professors can do to help our students develop their identities as professionals and their obligations to the greater society is to incorporate into the law school events that plug students into what’s going on in the “real world.“ I did just this in a small way this week by offering all the students the opportunity to attend and participate in a talk/discussion about the Supreme Court arguments that were heard last week in the DACA case.

The students were invited to attend a portion of my immigration clinic class. Food and pro bono credit hours helped, I’m sure, but the event brought a plentiful group of students I had not interacted with before, who were both knowledgeable about and interested in the issue of the day.  The event lasted only about 45 minutes, but that was long enough to produce a lively and I think informative conversation about oral arguments, professionalism, case theory, the role of policy, administrative law, and of course the specific legal issues raised by the case.

With so much of the law school endeavor focused on exam taking and other tasks that force students into a single-focused, competitive role, bringing them into a discussion about key issues at stake in our country in the moment could likely enhance their connections to their future and help them envision some individual goals they can aim for once out in that “real world.”

“Being Human To My Students And Letting Them Know I Care”

This is a wonderful blog post I found on the Institute for Law Learning and Teaching by Jane Korn, Professor of Law at Gonzaga University School of Law. As a current law student, I think that this practice should be implemented in all law schools for first year law students. I had a professor during 1L who did something similar. He would start the class every week with, “so how is 1L going?” and we could spend 10 minutes discussing general concerns about 1L and papers or exams we had coming up. Not only did it ease some of the anxiety, it also showed that the professor really cared about the students. It was like they were saying, “I’ve been there too and I’m here to support you.” Kudos to Professor Korn for setting aside some time in her class to do this!

“I have taught first year law students for a long time.  Please do not ask how long!  But years ago, I became worried about the mental health and stress levels of my first semester, first year students. I teach a four credit, one semester course in Civil Procedure during the first semester of law school.   On the last day of the week that I teach in Civ Pro, I take a few minutes out of class time and ask my students to tell me how they are doing.

The first time I do this, usually at the end of the first week of law school,  I tell my students that it is my custom, from time to time, to take time out from Civ Pro, and talk about anything they would like (with some limits).  In some years, it takes weeks for them to take me up on this offer.  Other years, they start right in.  They ask questions like the following:

  1. When should I start outlining?
  2. How much time should I spend studying every night?
  3. How important is getting involved in extracurricular activities?
  4. What if I don’t know what kind of law I want to practice?
  5. Do professors care about grammar and organization on a final exam? (I only answer what I expect and do not answer for other faculty)

I think that much of the time, they do not get a chance to ask a law professor these kinds of questions, and can usually only ask upper class students.  While we have faculty advisors, students may or may not feel comfortable asking them questions like the above.  They eventually do (and sometimes quickly) feel comfortable asking me a wide variety of questions.  They sometimes ask personal questions and, within reason, I answer them because it makes them feel more comfortable with me.  Questions on gossipy matters about other faculty are off limits. If for example, they complain about another professor,  I handle this question with a smile and say something like – you should ask that professor about this issue.

I set aside class time for several reasons. First, while I do worry about giving up valuable teaching time, lessening the stress of my students may make them more able to learn.  Second, students often feel like they are the only one with a particular concern during this first semester, and they often do not have the ability to know that others have the same concerns or questions.  In the first year, many of our students are not from this area and are far away from support systems, at least at first until they can make friends at law school.  The ability to know that other students have the same problems they do can lessen the feeling of isolation.  Using class time to answer questions to the entire group may help them with this sense of isolation and being the only one who doesn’t know something.  It also lets them see that their concerns are important and credible.

Every year my teaching evaluations reflect this process positively.  Students feel like I care (which I do).  However, the reason I do it is to increase their comfort during those first few exciting, confusing, and terrifying months of law school.”

 

The Legal Interviewing and Language Access Film Project (LILA)

By: Laila L. Hlass and Lindsay M. Harris

Teaching effective interviewing skills is a perennial problem. Although there are excellent texts on the subject, few examples of real or model interviews exist, particularly ones which incorporate collaboration issues between student partners, language access issues with the client, and how to address issues of bias when they arise in the interview.

In 2018, we designed, screen-wrote, produced and released The Legal Interviewing and Language Access Film Project (LILA), two instructional videos and a teaching guide featuring a law student clinic pair representing two different immigrant clients, in two different introductory meetings, one of which is conducted with interpretation.

Our goal was to better teach interviewing in our own experiential courses, but we also hoped to share this resource with our colleagues. Since the videos were launched, law school clinics and experiential learning programs across the country have adopted the use of the videos. At the time of writing, more than 100 educators at nearly 75 law schools have requested use of the teacher’s guide for these videos. This includes more than 30 immigration clinics, but also educators teaching in a variety of other clinics, purely doctrinal courses, as well as courses focused on client counseling and interviewing skills.

The videos raise a multitude of issues within interviewing including client-centered lawyering, collaboration, interpretation, and addressing bias. Our films enliven and deepen the learning environment by utilizing modeling, as well as stimulating classroom discussion, reflection and role play. 

In Interviewing Victor: The Initial Meeting, two law students Lisa and Max interview a teenage asylum-seeker in removal proceedings, Victor, raising a number of issues relating to initial client interviewing, including: Road mapping and organization of the interview; Building rapport; Confidentiality; Role description, including representation at later stages, and explaining the arc of case; Verbal and nonverbal cues; Tone; Answering client questions or ethical issues that are difficult and unexpected; Recording the interview and seeking permission; Taking notes; Form of questions; Word choice; Approaches to sensitive topics and response to client’s distress; Client-centered lawyering; and Working with a co-interviewer.

In Josefina: Using an Interpreter, two law students Lisa and Max working with interpreters to interview a monolingual Spanish-speaking client seeking a U visa as a victim of a crime in the United States. This video raises questions regarding: Using third person; Pacing of speech; Summarization and  expansion of interpretation; Challenges when one student speaks the client’s language but partner does not; Confidentiality; Use of interested parties, such as family members; Approaches to changing interpreters; and Use of common language words where the interpreter doesn’t know the intended meaning.

For faculty who hope to adopt the videos in a course, pro bono orientation or other training, please email either Laila Hlass lhlass@tulane.edu or Lindsay Harris Lindsay.harris@udc.edu for the teacher’s guide, indicating in which course(s) you are considering using the films.

Building A Solid Foundation Before Week 1

By Louis Jim, Assistant Professor, Albany Law School

One year ago, I began teaching Introduction to Lawyering, which is the required 1L course on legal analysis, communication, and research at Albany Law School. The textbook I used, like many “legal writing” textbooks, provided information about the types of legal authorities (primary or secondary) and weight of those authorities (mandatory or persuasive). And any textbook about legal authorities would, of course, also provide information about this nation’s three-tiered court structure. In class, I discussed those concepts, showed flow charts illustrating the structure, and distributed a map of the circuit courts of appeals. But I failed to assess whether my students truly understood the significance of the three-tiered structure and how that significance related to their other first-year classes.

This past summer, I attended the AALS New Law Teachers Workshop, where a number of presenters inspired me to think about new methods to assess whether my students understand the foundational needed to succeed in the first year and beyond. In response, I made two significant changes to my course design this semester. First, I required my students to complete weekly reflections in the last ten minutes of our Friday class.[1] The students must tell me two things they learned in my class and two things they want to learn more about in class. Students may then leave comments or ask questions on any topic even if the comments or questions are not related to law school.

Second, rather than simply discussing court structure with them, I created an in-class activity to assess whether students understood the significance of that structure. The students completed this activity at our first Friday session, which was the last day of their first week of law school. I rewrote a hypothetical that was originally written by my colleague at Albany Law School, David Walker, Assistant Professor and Director of the Schaffer Law Library, for a quiz in his advanced legal research class. A copy of the hypothetical can be found here:

The students spent the first ten to fifteen minutes of class reading the hypothetical. I then asked a series of multiple choice and short answer questions using Poll Everywhere based off the hypothetical. A copy of those questions can be found here:

I provided a link to the webpage where students would respond the poll’s questions, and students answered the questions using their laptops. Their anonymous responses were displayed on the large monitors at the front of the classroom. As we worked through the questions and hypothetical, I defined common terms that students would encounter in the cases they read for their doctrinal classes (e.g., motion, ruling, opinion, holding, judgment, etc.). I also distributed an outline that allowed the students to write the definitions and take other notes. A copy of that outline can be found here:

I hid the responses until at least three-quarters of the class had responded as I did not want a student’s response to be influenced by their classmates’ responses. By displaying their answers anonymously, every student could participate without fear of embarrassment, a fear prevalent in the first few weeks of law school. By using Poll Everywhere, the students who did not choose the right answer also saw that they were not alone. For each question, we also discussed each of the answer choices and why a particular choice was correct and the other ones were incorrect. Because everyone had to answer the questions, everyone—and not just the victim of the cold call—stayed engaged.

Because we completed this activity on the first Friday that we met, the students also completed their first reflection on that day. One student had commented in her reflection that she wished that we had completed that activity before the first week of classes began because it gave her a better understanding of the assigned case law in her doctrinal classes. I met with this student that following Monday, and she said she had a better understanding of her Week 2 reading assignments in her doctrinal classes after having completed the activity. Another student added that the activity filled many gaps in his understanding of the material in his doctrinal classes. Later that week, another student told me in person that she also wished we had completed the activity before the first week of classes.

As attorneys and/or professors, we often take for granted our understanding of the hierarchy of authority of the court system and our understanding of the terminology common in case law. Those just starting law school, however, may have never read a case before. But more often than not, the new law students’ first law school assignment requires them to read a case (likely more than one) and be prepared to discuss the case (or cases) on the first day of class. Those readings contain terms and concepts that new law students may have heard on television or read in a newspaper, but most new law students lack an understanding of how those terms and those concepts relate to the substantive law. Students may then feel discouraged in the first week because they don’t understand the concepts that seasoned attorneys take for granted. Although law students should and must develop skills in synthesizing rules and applying them, as educators, we must provide a solid foundation so that students can start developing those skills. With that in mind, next year, I hope to complete this activity even earlier so that students begin Week 1 with a solid foundation.


[1] This semester, I teach two sections of Lawyering, and each section meets once on Wednesday and once on Friday. On weeks in which we don’t have time to complete the weekly reflection in class, the reflection becomes an optional assignment that students can email to me. Much to my surprise and delight, some students completed the optional reflections too.

%d bloggers like this: