[game_edu] Evaluating individual students on semester long project classes

[Ian Schreiber]

Wow, what a great question. Always a challenge to figure out group projects.

One thing I've done that I found the most useful was complete "360" peer review, modeled on some of the personnel reviews I've been subjected to in the field (not that anyone likes going through these at work, mind you, but I can at least prepare my students for some of the things they might need to put up with later). First I have everyone, independently, send me a review of each individual on their team, at least one paragraph just saying what they thought each individual's contributions, strengths and weaknesses were, whether each person was pulling their weight, etc. - including themselves. I'd then hold a one-on-one meeting with each student and go over their own self-assessment together with my personal assessment and that of their peers (I would go ahead and collate and paraphrase everything ahead of time, keeping all comments general and not traceable to any individual peer). It's a little bit of extra time, made up for by the lower time burden
 of teaching a studio class in the first place :)

In my experience, students are usually a lot more harsh on each other (and themselves) than I'm inclined to be. And hey, I figure if students DO try to game the system and only say nice things about each other, I have to say that's (at least partly) good practice for the workplace: best to not go around badmouthing your co-workers, either to your manager or lead or to others outside the company. These things follow you around, and if the students can learn that in a capstone class, I think that's a great lesson and would use the time in the individual meetings to make sure they see all the implications.

A lot depends on whether the course is earlier or later in the curriculum (some programs have a smaller project-based course in sophomore or early junior year, then a year-long larger project in the final year). The point of the earlier course is to get the students used to a team environment: multidisciplinary team skills, communication skills, project pipeline, stuff like that. For such a course, I'd grade more heavily on process and ability to get along, with less emphasis on the quality of the final project. For senior-level projects, though, I'd want to see something impressive.

One dilemma: several folks here have talked about measuring "effort" as opposed to results. I think we need to be clear here about not just what to measure, but why.

Do you grade effort? On the one hand, yes, you want your students to put in a certain amount of work. On the other hand, if a student "works smarter" - finding clever ways to implement the project with less time spent - should they be penalized in comparison to a student who is just not demonstrating a high level of skill, but dutifully bangs their head against the keyboard for ten hours a day? Or do you just grade the results of their work, regardless of how much time they spent, and if they put in less time and got better results then more power to them? I understand the importance of imparting a good work ethic... but at the same time, I'd worry that we'd be sending the wrong message by placing the metric of hours worked on a higher level than quality of work. (I could even put this in gaming terms. Do you favor a grading system like an RPG which rewards grinding in lieu of skill, or one that's more like Chess Elo that rewards results only? Or
 somewhere in between the two, like Matchpoints in professional Bridge?)

Ultimately, I favor grading the result of the project itself. If the students put in the effort, it generally shows in the final work. As for who contributed to what (so if some parts are polished and others are rough, how to tell who to reward), the students all know full well who the MVP and who the deadwood is, even if you don't, and focused peer reviews that ask students to identify specific contributions can ferret that out pretty easily.

- Ian



________________________________
 
On 1/4/12 10:09 AM, "Robert R. Kessler" <kessler at cs.utah.edu> wrote:


>The techniques that we've tried are (note, we typically have an area lead

>appointed over engineering, arts, and design, and then a team lead)

>

>1) Have each area leader report whether each student did their work and

>how well they did it (we've tried evaluating them as: C - if they did

>their job, B - if they did it and did an excellent job, A - if they did B

>work and then went above and beyond and did more - as in worked on extra

>items from the sprint backlog).  For the area leads, they are evaluated

>by the team lead and then the team lead is evaluated by the area-leads.

>

>2) Have each student create a gamer blog and include in it each week,

>what they were assigned to do.  Then make an entry at the end of the week

>with what they accomplished and to show evidence.  We the teaching staff

>then go in and evaluate it weekly.

>

>Neither of these have been quite satisfactory.



_______________________________________________
game_edu mailing list
game_edu at igda.org
http://seven.pairlist.net/mailman/listinfo/game_edu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://seven.pairlist.net/pipermail/game_edu/attachments/20120105/bce8531d/attachment.html>