Friday, November 20, 2009

Why Don't Evaluations Incldue Exams?

We've been thinking about this for a little while, with course evaluations this past week - don't forget! - it seemed to be the perfect time to post it.

UVA certainly deserves some credit for allowing students to do course evaluations, and making the evaluations available to everyone. Of course, some professor fail to inform their students that such evaluations are available, and some students use the evaluations as a platform for lengthy screed against the class, but for the most part, course evaluations are a useful tool.

Or, at least they have the potential to be. The problem is, the evaluations never speak to the most important aspect of any course: the final exam. Because the exam typically will account for 100% of your grade in the course (and let's face it - everyone here, gunner or not, would prefer having good grades to bad ones), it's incredibly important for students to know what the exam is like.

For example, is it fully open book? Closed-book? Multiple choice? Take-home? More importantly, we think, is whether or not the exam was a "fair" one. Did it test material not covered in class? Was it a single sentence long? Was it the type of exam where you had time to develop your thoughts, or was it an absolute horse-race where you were pounding away the entire time? - see, these are things students should know.

It's also worth noting that many students would be interested to know whether or not a professor is perceived as being a "fair" grader. We don't want to open up the Pandora's Box on what, exactly, this might mean, but it's known to most people around UVA that some professors have the reputation of being rigorous, thorough, and fair graders. And we have heard complaints of Professors who do the opposite. Maybe this aspect is a bit more problematic than the above, but it bears mentioning in a discussion of more informative evaluations . . .

Anyway we are, of course, aware that part of the reason for the current state of affairs is that the honor code doesn't allow you talk about most exams, even after you and everyone else in the class has taken them. (Do this, and you will be kicked out of school - in theory). But that's something that should change - grades in law school are just too important for the exam system to the "black box" that it is now, we think. We understand that this could mean that professors can't recycle exams (or if they do, everyone will know about it) - or maybe the comments could be limited in some way. It's just frustrating to me that students have to go into this process - probably the most important of their (very expensive) law school career - completely blind.

One possible ameliorating alternative would be to make all professors post at least one exam that they have given in the past on LawWeb, along with the instruction sheet. While this wouldn't allow students to comment on whether the perceived grading was fair, it would allow students to have a good guess at what form the final evaluation will take before they sign up for the course. But we still think, ultimately, evaluations and the honor code should be reworked to include exams.

Anyway, as always, we welcome your thoughts - before, or after exams.

38 comments:

Anonymous said...

Pro tip: law school is curved. There is never any reason to provide information to the students re: performing well in law school, because it is impossible for them to systematically improve. To the extent you make information more readily available, you're just penalizing the ambitious.

Anonymous said...

Good post. Another question related to the black box that is law school grades: why in the HELL can't they post online the class grade distributions?? Why do they have to be valcroed to a table in Dean Bennett's office?

Anonymous said...

Actually, the Honor Code isn't standing in the way of this at all. Rather, it's SRO. The Code simply says, in effect, that you must follow professor (or in this case, SRO) instructions. If SRO instructions permit some kind of discussion/evaluation, there is no honor problem.

In terms of preventing excessive disclosure, there's no reason that there can't be some kind of supplemental evaluation that students complete, which would consist solely of button-style ordinal ranking "great, good, fair, . . ."

Anonymous said...

We never see our exams, so any comment on grading would basically amount to, "I worked really hard and got a B. This is bullshit!" or "I grabbed an SBA outline and got an A, this class is soooo easy." And with the curve, those types of comments are totally meaningless.

I do like the idea of requiring old exams to be made available, at least if the prof has ever taught the class at UVA before. I think most profs wouldn't mind -- they may not want to do *extra* work to make exams available now but if they "had" to, they would. ("Had" in quotes b/c with tenure, it's allll optional.)

And putting grade distributions online is a no-brainer. I bet SRO could do it in 10 minutes, but nobody's ever asked...

Anonymous said...

I think the post-exam radial buttons and online curve distributions are great ideas.

On the same general topic, it would be nice if the current survey options were a little more fine-tuned. For instance, they ask how many classes you attended and 80-100% is a single option--I've only seen someone choose an answer other than that once or twice. Likewise, I think people hesitate to rate a professor "average" when the attached percentage range is 40-60% (if I'm remembering right), which probably encourages "rating inflation." Maybe just make everything a 1-10 scale?

Anonymous said...

Why Don't These Posts Incldue [sic] Spellcheck?

Anonymous said...

It would be nice to know whether the exam is closed book or open book before hand. Other than that, I can't see how an exam can be more or less fair.

Anonymous said...

I get your fair grading point. I got screwed by a professor the seemed to just give out grades randomly. When I asked him about it, he just gave some vague explanation of why I didn't do as well as other people. Funny thing is he gave the same explanation to everyone, regardless of how well they actually did. Unfortunately I was forced to take his class (a 1L class) so evals probably wouldn't have made a difference.

Anonymous said...

The lack of transparency on grades (among some - not all - professors) is outrageous. And it approaches negligence that the law school seems totally uninterested in addressing it.

Anonymous said...

Agree with most of the comments above. It also seems unfair that 1L professors have such a wide range of grading policies. Some will give 90% B+'s and scatter a few A- and B's. Others will have several B-, C+, and even Cs. Given the job market and how seemingly important grades are it doesn't seem right that some 1L's, who cannot chose their professors beforehand, are stuck with professors intent on giving C's, while other can ride the curve and safely get a B at the worst. Perhaps some more uniform guidelines other than "B+ curve" is warranted.

Anonymous said...

good point, 9:55. The standard distribution should also be regulated, like the curve is. The SD should be very small -- i seriously doubt profs have the ability or the inclination to accurately finely tune their grades so as to give many Bs, B-s, A's, etc.

Obviously, there will be a few true standouts (your A's, A-s), and a few really horrible exams (B's, B-'s), but 90% of the exams likely look identical.

Profs should also be forced to justify their grades -- in other words, if there are enough complaints about arbitrary grades (one prof comes to mind), Dean Bennett should be allowed to audit this prof and receive a full explanation/review of the grading criteria, etc.

UVA (and law schools in general, except YLS) apparently wants it both ways -- they want grades to be so critical but they don't want to invest the resources into ensuring a fair, consistent system.

Anonymous said...

I understand the sentiment for a uniform curve (say, 30% A-, 40% B+, etc.) rather than just a mean. Many law schools do this. Certainly, if some profs are giving out Cs and C+s (and A+s) and most aren't, that's a problem. (Although at least one 1L prof I had apparently gave out Cs as punitive for missing too many classes -- as an alternative to failing, that's not so bad!)

But here's the counter: Let's say Professor A has an exam where 90% do, in fact, look the same. He ought to be able to give out 90% B+, 5% A- and 5% B. Professor B has an exam where about 1/3 of the class did fantastically, 1/3 of the class did terribly, and the rest were just mediocre. On a strict curve, at least one and possibly both of these profs would have to make fine-grained distinctions that may be unwarranted. I'm not sure how that's any better or less artificial than letting the prof set his own curve shape.

Possible solution that might be overly complex: Maybe something like, 75% of the grades must be [A-,B+,B], 10% must be outside that range (mostly B- and A), and the remaining 15% can be assigned at will as long as the overall hits the 3.3 mean.

Anonymous said...

If you start "requiring" that a certain percentage be outside the A-/B+/B range, you are making the process even more arbitrary than it already is. The debate here underscores the main issue: law school grades are largely guesswork. Who knows what makes separates a B+ from an A-, or a B from a B+? And yet employers put a crazy amount of emphasis on them. This creates the cut-throat environment that many detest about law school (and yes, despite our "laid back" image, this environment exists at UVA. Just look around the library next week and you'll see it!)

What's the solution? Really there is no good one. To me, if you're going to have grades count for so much, there should be at least some objective measure (multiple choice tests, assignments throughout the semester, etc.) to go along with the highly subjective traditional law school exams. But this is so contrary to the status quo that I doubt any faculty would go for it (and it would be far more onerous to them time-wise, with all the extra grading).

Anonymous said...

In the past, I don't the arbitrary nature of some professor's grading mattered that much. I'm sure there were a few extreme examples where someone who normally got stellar grades got screwed out of grading on to law review because of some asshole professor who graded randomly and gave them a B.

In this economy though, every grade matters. Getting average grades may keep you from getting a job this year. I have not found a job yet and have pretty good grade (top 25%), but I believe they would have been a lot better if I had the opportunity to choose my 1L professors (at least for one class in particular). The good news for 1Ls is that the professor I am referring to is not teaching 1L classes anymore.

Anonymous said...

Oh poor victimized you - you had Doug Leslie. Of course I could point out that you only had to study for 3 exams. So contrary to your assertion about doing better, you likely would have done a lot worse had you taken 4 REAL exam classes (like, say, all your classmates).

Sorry, was I mean? Maybe you can run to the Dean to complain about how insensitive I was.

Anonymous said...

8:48 - i'm going to assume they aren't talking about Leslie.

their complaints are about *arbitrary* grading.

there was nothing arbitrary about the way that asshole graded.

Anonymous said...

"So contrary to your assertion about doing better, you likely would have done a lot worse had you taken 4 REAL exam classes (like, say, all your classmates)."

Flawless reasoning here, Leslie groupie.

I'm sorry you won't be able to take the 5 Leslie classes you were planning on taking to fraudulently boost your GPA. It's a shame b/c you won't do well in the honestly graded classes you have to take instead. Because you're stupid, see?

Anonymous said...

1Ls who had Leslie are disproportionately represented on LR.

Anonymous said...

Not sure your problem with 8:48's reasoning 9:54. All things being equal, studying more for a final leads to better grades. Sure, Leslie's grading might have been arbitrary, but that works both ways. Since his students wrote that exam off anyway, they had more time to study for other finals.

Anonymous said...

10:56 - Those students who had the advantage of not having to study for an exam had to compete (mostly) against their sectionmates, who had the same advantage. And there are just too many variables contributing to exam performance to make such a statement, especially the way it's worded by 8:48

Anonymous said...

11:16 -- not really. I'm assuming their section made up about 1/3 of the other three classes... with the 2/3 remaining comprised of people with four exams. Seems like an advantage, albeit a slight one.

Not true about Law Review, though -- Sections A, B, and G were all under-represented on Law Review this year (c/o 2011).

Anonymous said...

Doubt risk of getting a randomly (or not so randomly) low grade is made up for by the extra day of study time for the other exams. Law school needs a system of auditing grades. If there are enough complaints, have another professor look at the exams and see if the grades are justified or not.

Anonymous said...

But Leslie's grades include "participation" so auditing wouldn't really be appropriate to his "style".

Of course, we could just outlaw participation grades, but then gunners would actually have to face the fact that nobody cares about their hypotheticals.

Anonymous said...

Auditing, whatever - there needs to be SOME sort of process for challenging grades if professors are going to be so un-transparent.

On a side note, does anyone remember how they released copies of Obama's con-law exams and answer keys during the presidential campaign? Can you think of any professors at UVA who are that responsive to questions about grades (read: not lazy)? And he had three jobs at the time.

Anonymous said...

A bigger problem is that classes aren't curved on to a strict 3.3 average. Some 1L professors such as Professor Bonnie decided to curve their class to a 3.26 whereas other professors curved to around a 3.35. Most professors were slightly above a 3.3. At least for 1L classes, there should be a strict requirement that the professor curve to a 3.3.

Auditing
Seems like a bad idea to me. Except in the case of pure miscalculation.

If you are suggesting a faculty review of other professors' grades then it will accomplish nothing. Let's say that all of the B & B- students complain (which are really the only students that will), then will another professor review the original exams and give them a grade, or merely look at the the original method? Either way, the auditor isn't going to want to step on the toes of the original faculty member. Further, isn't it far more arbitrary to have another professor determine what is or isn't fair given that the original professor taught the course.

If you're suggesting that the original professor only needs to show their method, then every professor will pass this inquiry. During my 1L year, I asked all of my professors about what their grading method was for the exam (i.e. whether they used grids, did a straight point count, normalized each question and curved it, ect.) and all of them told me how they scored exams.

Finally, Cary Bennett would not be qualified to evaluated whether a system is fair or not. Although he's administered exams for 30 years, he has neither taught at a law school nor taken any law school classes.

The real problem isn't that professors need to be audited, it's that students don't know whether they're any good at law school until they've taken exams. Having taken practice tests with other students, it's clear who understands how to write a test and who merely understands the law.

Anonymous said...

Auditing could work if done right. Grading really shouldn't be that subjective. If a professor has a methodology he should be able to explain to another professor in the field. Sure there will be disagreements, but if there are blatant disparities then they can be detected. The problem would be discovering when there is a real problem and when it is just a bunch of whiny law students who aren't happy with their grades. But we're a law school, for crying out loud. It can't be any harder than anything a judge does in determining whether there is sufficient evidence to go to trial. Have a process where if you can show that there is a reason to suspect that the grading is arbitrary, have another professor (or better yet, a panel of professors) audit the grades. Probably would never need to use it, but at least it would be there to make sure the system is more fair.

Anonymous said...

is leslie really going to be allowed back - whether teaching 1L's or anyone else? the guy's a walking lawsuit; i can't believe they'd bring him back.

Anonymous said...

never mind, his uva webpage is up and indicates he is still teaching. amazing. i was pretty sure when i left uva last spring that lesliegate would result in his firing, but i guess the admin are bigger cowards than i thought.

Anonymous said...

You try firing a prof with tenure because some 1Ls got their feelings hurt. Even if he is clearly crazy.

Also -- bitching about grades as a black box? What about the hiring process? Total black box... people with better grades *tend* to do better, but that's a very rough correlation, and otherwise there's no "objectivity" there. Or did you go to law school to avoid the real world?

Anonymous said...

What's up with the stupid waitlist process? I just got dropped from all my waitlists because I was traveling today and wasn't able to get online to "confirm" my spot. That's the second time that's happened to me. Such a stupid system!!

Anonymous said...

"...part of the reason for the current state of affairs is that the honor code doesn't allow you talk about most exams, even after you in everyone else ..."

you in everyone else? Seriously?

Rule 12 (f) said...

O M G - a typo!

Anonymous said...

"because some 1Ls got their feelings hurt." Yeah, that's all Lesliegate was about - "hurt" "feelings".

Anonymous said...

After professors grade exams, do they match up blind grading numbers to student names and then submit that info to SRO? Or do they submit the blind grading numbers to SRO and SRO matches the blind grading numbers to student names?

Anonymous said...

Good question, 10:10 - any answers out there?

Anonymous said...

er, 1:10

Anonymous said...

1:10, this doesn't really answer the question but I know that professors are aware of who did well in their class.

When asking for a reference, the professor knew that I did well without me having to mention my grade.

Anonymous said...

This is the process: Exams are graded; those grades are matched to BGNs; Profs can then adjust grades slightly in order to fit the curve (which, if they do at all, they usually do based on class participation). Obviously, they can't do the last step until they see the names of the students matched with the grades.