As law schools engage in outcomes assessment a key question involves how to collect institutional data on student achievement. In A Simple Low Cost Institutional Learning Outcomes Assessment Process, I suggest one way to engage in data collection that requires relatively little additional law faculty time, relatively minimal expense, and does not require faculty to change how we teach or assess in our own courses.
Law school institutional learning outcomes require measuring nuanced skills that develop over time. Rather than look at achievement just in our own courses, institutional outcome-measures assessment requires collective faculty engagement and critical thinking about our students’ overall acquisition of the skills, knowledge, and qualities that ensure they graduate with the competencies necessary to begin life as professionals. Even for those who believe outcomes assessment is a positive move in legal education, in an era of limited budgets and already over-burdened faculty, the data collection necessary to engage in the new mandated outcomes assessment process raises cost and workload concerns.
To address those concerns, the article describes a process being used by Georgia State University College of Law [GSU COL] to collect institutional learning outcomes data. GSU COL has developed a rubric method to assess a wide array of learning outcomes.
We modeled our process on work being done both by the American Association of Colleges and Universities [AAC& U] Values Rubrics Project and medical educators’ Milestones Project . Those educators use rubrics to assess a wide range of nuanced skills such as critical thinking, written and oral communication, problem-solving, intercultural competence, teamwork, and foundations and skills for life-long learning.
Below, I briefly describe GSU COL’s process for collecting institutional learning outcomes data.
The Institutional Data Collection Process
After identifying our institutional learning outcomes, we developed a five step institutional outcomes assessment process to collect data from GSU COL faculty. The faculty data focuses on law student performance in various courses.
- Draft rubrics
First, we engaged our assessment committee and, in some cases, ad hoc faculty committees, in drafting rubrics. The rubrics had to be general enough that they were usable across a wide range of courses and adaptable to various types of course assessments. To draft the rubrics, we looked to our own experience and other sample rubrics such as those developed by AAC&U and medical educators as well as those developed by legal educators. The article’s appendix contains GSU COL’s draft rubrics for our eight learning outcomes.
- Pilot test rubrics
Second, we identified courses that would use the rubric – courses where the skills being measured were already being assessed. For example, for the basic legal knowledge and analysis outcomes, we chose first year and upper level doctrinal courses. For self-reflection and client interaction outcomes, we chose clinics, etc.
We are pilot testing each rubric with faculty who will use the rubric and using their feedback to refine the rubric. Because our assessment process is cyclical, each year, we pilot two rubrics and use two rubrics for actual data collection. Thus, our rubric development process remains a work in progress and it engages a significant number of faculty members. This helps ensure validity, engages faculty outside the assessment committee, and, hopefully, builds faculty buy-in.
- Use the rubrics
Third, every year, we ask faculty in designated courses to assess and grade as they usually do, adding only one more step – completion of a short rubric for each student. Most faculty members have said this process adds very little additional time to grading.
Given our different outcomes and the cyclical nature of our assessment process, each year, different faculty will use the rubrics. For example, one year, legal knowledge and analysis rubrics will be completed by doctrinal faculty. The next year, legal research and writing faculty as well as seminar faculty who assign papers will complete rubrics focused on legal research and writing, etc. Thus, we spread the workload and engage as many faculty members as possible in the institutional outcomes assessment process.
- Enter the data
Fourth, we enter the rubric data from each course into a computer. This data entry process can simply involve an Excel spreadsheet, an SPSS program, or it can be more complicated. For example, we worked with GSU university computer programming graduate school GRAs to develop software compatible with the university computer system to allow us to manipulate the data in numerous ways. We currently are working on developing the software program so that it can be used by other institutions.
- Use the data to analyze student learning and make changes if necessary
Finally, we are using the data to prepare reports about institutional level student outcome achievement. In order to increase the validity of our findings, our reports contain information collected from multiple sources. For example, for each institutional outcome we have data from the rubrics faculty complete and externship site supervisor evaluations. Additionally, LSSSEE survey data has information relevant to many of our outcomes. The results from all that data are included in the faculty learning outcomes assessment report.
This Fall, our faculty will discuss our findings on the first two outcomes we measured – legal knowledge and legal reasoning and analysis. While I noted at the outset that the data collection process does not require faculty to change how we teach or assess, our discussions in light of the data we have gathered may lead us to collective decisions that some of us will adjust our teaching and assessment processes in an effort to improve student learning. That is the entire point and purpose of the learning outcomes measurement process. However, before we can begin that work, we had to figure out how to get information that allows us to have informed discussions. The steps summarized above, and described in more detail in the article, are one way to do that.
Other resources
The data collection method above can be used both to measure institutional and even course level learning outcomes. However, multiple ways to collect data exist. Other good resources with concrete data collection methodologies include Andrea Funk’s excellent book: The Art of Assessment, and Lori Shaw and Victoria Van Sandt’s seminal work, Student Learning Outcomes and Law School Assessment.
Filed under: Uncategorized |
Best Practices for Legal Education 