One of the outcomes of the Harvard budget crisis is that the budget for Teaching Assistants has been brought into line. At Harvard, we've been ridiculously spoiled with TAs for many years now, with a TA for roughly every 12-15 students; last year, I was able to get 6 undergrad TAs for 80 students in my Algorithms and Data Structures class. This year, I'll have a more reasonable TA for every 18-20 students or so. Yes, I know that remains quite luxurious compared to many places; I'm not complaining. But it is a change I have to deal with.
One of the big responsibilities for my TAs is grading. I assign a fairly substantial load of homework in my undergrad class, and unfortunately the number of problems where you can just check the answer without reading through the student's work is small. (They often have to prove things.) And even checking answers takes a long time. Given the new TA/student ratio, it seems unfair (and, quite frankly, unworkable, assuming TAs stick to anything close to their supposed working hours) to have the TAs grade the same homeworks in the past. So, it seems, I'll have to change something. The natural options seem to be:
1) Reduce the assignments. Take what would in the past have been the hardest problem on each assignment and make it an ungraded "challenge problem", for instance.
2) Change the scope/style of the assignments. Less proof-type work, more numerical exercises where you can just check the final answer.
3) Introduce probabilistic grading. Only 5 of the 6 problems will be graded -- you just don't know which 5 in advance.
4) Allow people to work in pairs for assignments. Fewer writeups to grade.
5) Grade more myself.
The first four options all have fairly big negatives associated with them. (Actually, I don't have a problem with probabilistic grading, but I have no doubt it would cause bitter complaints from the students, even if I gave a lecture explaining why it was a reasonable approach. There would always be students coming up afterward to complain that they would have gotten a better grade if I just graded all of their problems, and I don't look forward to having to explain the policy to higher-ups. And working in pairs isn't necessarily a negative, but they can already talk about problems together, and they can work in pairs for programming assignments; I think they should practice writing up proofs themselves.)
The main downside to the final option is, of course, to me personally. I do, already, do some of the grading. It's very time-consuming. Even if the time per assignment is small, multiply by the number of students (I expect 60 this year) and we're talking a good number of hours.
So how much grading should a professor do? How much are others of you doing? Or does anyone have other creative suggestions for solutions before the semester starts?
Subscribe to:
Post Comments (Atom)
24 comments:
I know as a TA at what you could consider a smaller school, I have about 50 students whose assignments I grade, but it's a math class, so while the majority of the problems have work, while others have little.
I suppose the culture among the grad students is a bit different there, but I would say that I don't find the amount of work I'm given unfair; I enjoy it! I like your idea for probabilistic though, that's the first time I've heard of that. I could understand why it would be a hard sell though.
This is a bit science-fiction but you may require the students to write their proofs using a proof assistant. To avoid the burden of the details of formal proofs, you allow them to assume boring/tedious lemmas and focus on the high-level intuition of a proof. That way, you can still ask for proof-work, and check those in no time using a computer.
Group homework. Each student writes up their own solutions, but students work and submit in pairs. For each problem, one of the pair is chosen at random and graded, and both students receive that grade. But no credit for pairs that submit identical solutions; each student still has to write up her own proofs herself.
As for the complaining students, here's how I respond: Yes, you would have gotten a better grade if we'd done something else, but we didn't, so you didn't. For any two grading systems X and Y, there are always students who prefer system X than system Y, and other students that would prefer system Y than system X. But we had to pick something, and we have to be consistent; changing the grading standard now would be grossly unfair to everyone. Also, stop worrying about your stupid grade and learn the course material!
For the record, I have three TAs this semester for a class of 120+, with short weekly homeworks. Students can submit solutions in groups of up to three (one solution per group; no randomization). In practice, this will probably mean about 50 homework groups. I can usually hire one or two undergrad graders to absorb some/most of the written grading burden. We also have a rolling oral homework system: Each week, 1/3 of the groups each get 30 minutes to present their solutions on a whiteboard, 10 minutes per problem, with the presenter of each problem chosen randomly from the group. (Everyone in the group gets the same grade.) The oral homeworks are a huge time sink, but it pays off.
In the past, I've asked huge classes whether they'd prefer (1) letting us grade half of their homework and getting it back quickly, or (2) making us grade it all but getting it back much later. The students have always chosen (2). Even when it meant that the last third of the homework wasn't graded until after the final.
A grad TA position is nominally 20 hours per week. Do you really assign so much homework that it is an hour per week per student for the TA to grade?
Grad TAs usually do a lot more than just grade homework: they run recitations, answer email questions, hold office hours and make/grade exams. So why do you assume they are just grading hw?
Erik had an interesting grading scheme for his Advanced Data Structures class at MIT: each question had just three possible grades 0, 1, 2. Also there was one page limit for each answer. The TA got back to us fairly quickly so it must be less work.
At Yale I TAed Algorithms twice. Each time there were approximately 40 students taking the class, and there were 2 graduate student TAs. Algorithms had weekly assignments which seem similar to what you describe giving. The TAs would alternate grading assignments, and the workload was pretty reasonable for the graders.
I don't know if undergraduate TAs are expected to put in the same amount of time as graduate student TAs -- at Yale undergrads aren't allowed to grade other undergrads, so all the grading was done by grad students. But we graded every problem, and the professor teaching never had to grade homework.
The slow part of grading is figuring out what students are doing wrong.
Letting students work in small groups and write up answers separately is good, because this allows the graders to look at all answers to figure out what the group was trying to do. It probably also decreases the number of wrong answers, making grading go faster.
It is also often possible on open-ended problems to add in a small subquestion that a student with the right idea should almost surely be able to answer correctly. This allows the grader to decide quickly if the student has the right or wrong idea.
Might be Harvard students have higher expectations -- but in a typical big-ten university in the corn-fields, it is completely standard practice for graders to grade randomly sampled ( < 10 %) of home-work problems (say in a 400+ size freshman calculus class).
Make students evaluate each others paper. TA will score each student on two fronts - the paper and the evaluation of other students. This makes students critique others work, which I think is a valuable skill.
Jeff -- I appreciate your suggestions. Again, I have my issues with group homework -- I do already have students work together on programming assignments, I'd like them to do some of their own assignments (particularly proofs) -- but in a resource-constrained environment it makes solid sense (as does probabilistic grading).
As for the second paragraph, I do take the same basic approach with complaining students. I hope it's more effective for you.
To Anons 4+5: A standard TA assignment, grad or undergrad, is typically "quarter time" or 12.5 hours per week at Harvard, not 20. And as Anon 5 points out, my TAs hold a weekly section, hold 2 office hours per week, answer questions by e-mail, and do other course administration. Really, I don't expect them to do any more grading than they already do.
Anon 6: I like Erik's approach for a grad class; I'm not sure how it would work for an undergrad class (but it may be worth a try!). I really like the 1 page per answer limit; I think I'm going to invoke that.
Can you reduce the number of graded homeworks (or get rid of them altogether)? While doing homework is the best way for the student to learn, I find it is not the best way to assign grades (since cheating is so easy). So why not post optional, ungraded homeworks (with full solutions posted a week later), stressing to the students that the homeworks will be great preparation for the exams. I would bet that writing up detailed solutions will be less work (and more enjoyable) than grading. And students who don't understand the solutions can come to office hours...
You (and professors in general) should grade more homework. This allows you to see what the students are learning/doing and evaluate whether or not you're an effective teacher. Furthermore, I believe each professor should keep track of how time-consuming their homeworks are. It is irresponsible to assign too much work because a professor feels that his/her class/discipline is somehow more important.
I'm going to disagree with the last poster a bit: I think it's important to strike a balance with the amount of grading you do. This last semester, I shared the grading load for homeworks approximately equally with my two TAs and one co-teacher (yes - we're spoiled at CMU as well, but in fairness, it was a new course we were developing). My TAs handled the project grading. I found this split a bit more than I would prefer: I'd rather have sat down with my TAs, graded a few assignments end-to-end with them to make sure we were all on the same page, and been done with it. As it was, the homework grading took away a bit from things that only I could do as the professor -- course & lecture development, etc. To the last poster: It's all relative. Had this not been a new course, I'd have been happy doing more grading, but the reality is that we (faculty) have a finite time budget for our courses, and there are better and worse ways to allocate that budget.
I'm actually a fan of the peer-grading system. David Karger used this for the (grad) algorithms courses I took, and I thought it was great. It didn't take too much time, and it forced everyone to learn the correct solutions to the problems, and to think about what was wrong with the incorrect solutions. But it can be a risky approach if grades actually matter, as opposed to in a grad course.
As a professor, I have no objections to grading a random subset of problems. I'm pretty sure our higher-ups would support any reasonable grading policy as long as it was clearly spelled out in advance to the students. I've come to believe that these are really only two things that are mandatory with homework: Clear requirements and rapid turn-around. My new years resolution for a few years running has been to reduce the homework turn-around time for every class I teach. I think I may try to do that with my ugrad distributed systems class next year by using randomized grading + having the TAs go through the un-graded problems in recitations - possibly with peer grading. :)
Thanks for asking this question - it's good to hear people's viewpoints about it.
Another anonymous in a big state school here (not in the corn fields) who has always done probabilistic grading and never got a complaint.
I once attended a course, by Alexander Postnikov, where the homework had 9 problems, each worth 10 points, and your score was taken mod 30 (though 30, 60, 90 counted as 30, not 0).
Your initial post was misleading. Your old TA budget looked positively luxurious but at 12.5 hours per week each the budget looks pretty normal and the new one does seem tight.
It is not easy to answer your question since it is hard to judge how large the workload is for students in your class. Sometimes we can get the same benefit from actually assigning fewer problems, even though there are ones that we like and want to ask. I handle this by including "extra credit" problems. These allow me to ask some of my favorite questions targeted at the more motivated students without having a large segment of the class incented to answer them with confusing drivel when they don't really have a good answer. This saves a bunch of grading time.
(Eztra credit problems are not required to earn a top grade but top students invariably want to work on them.)
I currently teach with a ratio of about 70-75 students/TA. Yes, that's right: over a factor of 3 worse than yours.
My view is that you should not change the problems you offer; in an algorithms course, thinking about great problems is one of the most important ways you learn. Instead, you should allow the quality or quantity of the feedback you offer, and the accuracy of the grades, to suffer. If you have to choose between "great problems" and "detailed feedback/accurate grades", continue to prioritize great problems.
You can do probabilistic grading. You can just tell the TAs to grade less carefully. A third option is to hire undergraduate graders: not TAs, people who are hired only for the purpose of grading homework. You can probably hire them for $10-12/hour, so they're a lot cheaper than TAs. Allocate a small fraction of your TA budget (say, 10%) towards undergrad graders. Have the undergrad graders help grade the homeworks. Yes, they're a lot less accurate in their grading and the feedback they give is not as good, but on the other hand it enables you to grade every problem.
Thanks to everyone for the interesting and worthwhile feedback. I'm definitely going to ponder probabilistic grading. I must say, I can't recall ever being in a class that used it (I was a Harvard undergrad...), and hence my original opinion that students would revolt. Maybe it's not so bad.
I've become reasonably attached to peer grading: assign a few students to grade under the supervision of the TAs each week. Typically, each student grades once, and grades only one problem (so they can become an expert on it). I reserve 5% of the course grade for grading, to provide the proper motivation.
The students aren't set free to grade independently---rather, they sit in a room and work beside the TA, who is available for initial discussion on how to grade the problem and also for consultation on unusual answers. So the TA is still involved, but their involvement scales to a larger class. This has been my approach for several years and it seems to be working.
My courses generally involve final projects, and I grade those myself.
I completely agree with AnonProf's tradeoff: Great problems are the heart and soul of a good algorithms class.
I have my issues with group homework
Yeah, me too. But at some point it became a practical necessity. One result that I expected when I added group homework many years ago is that homework grades went way up, and the first midterm average went down. Surprisingly, though, later exam averages actually went up.
Another strategy I use to simplify grading is giving 25% credit for writing "I don't know" (and nothing else) for ANY question, and absolutely no partial credit for regurgitation and/or logorrhea.
As for the second paragraph, I do take the same basic approach with complaining students. I hope it's more effective for you.
It works reasonably well for me, actually. Students here are generally accepting of the fairness argument. But probably my students are also less used to perfection than yours, and they certainly pay less, and so may feel less entitled. Also, I am mean.
David -- Thanks for the explanation. That sounds interesting. I'll ponder it...
Jeff -- Are you trying to say I'm not mean? Because there seems to be a number of students who would tell you otherwise...
When I taught Complexity at Berkeley, I took an even more extreme approach than giving credit for "I don't know." I was giving full credit for the statement "I know this."
This seemed to work well coupled with the warning that grading is not a linear function, and underperforming on simple questions during exams will erase the "I know this" credit.
It also seemed to diminish complaining about homework grading, since everybody understood they can get max on those, if they wanted it.
Of course, this system assumes maturity and failed miserably for a few students. I guess if colleges are seen as places to grow up (in the US), this might not be too bad an outcome :).
why not outsource it, let the grad students concentrate on research
Post a Comment