2010 IGF Finalists: Main Competition

I can definitely understand certain games receiving multiple nominations (it happens in film and music awards all the time). However, I get the feeling that certain games weren’t really evaluated based on specific criteria for each category, but that the judges enjoyed the game as a whole and awarded high scores for all categories.

The audio category feels especially random this year as compared to the past, where many of the finalists featured audio that was far more central to the game (either the music was phenomenal, the implementation was unique, or there was something that really caused the audio to stand out). This year, it doesn’t seem to be the case at all. The audio in these games is good and fits quite well, however in most cases I don’t see how the audio was “excellent, innovative or impressive”… it was just good. There were many other contestants this year whose audio truly stood out. Looking at these comparisons, it’s really difficult for me to believe that the judges actually evaluated the audio as a separate category.

I hope in the future, more consideration will be given to specific criterion in each category, and that judges don’t just give high marks for all sections if they like the game as a whole.

“But I stand by our voting process – it’s the aggregate of all those judges, and that makes it fairer than one opinion, ultimately.”

nobody’s suggesting only one judge, though; that isn’t what the objection was.

the theoretical way i’d ideally structure it is to find 20 or so judges willing to play every single game, and give them more time to judge all the games. so individual games would still be judged by as large a variety of people, it’s just that it would be the same people, rather than a random set of people. i don’t think that’d be any less fair or less of an aggregate than a dozen or so judges at random out of 150 judges playing a game.

but anyway, since the igf has been using its own strange process so long i don’t really expect it to change, i mainly went into it to explain why the results are what they are.

also, another thing is it’s not just people having personal favorites that they are upset about, it’s about large numbers of people expecting a results that at least marginally reflect the community’s tastes. i.e. there are even people who don’t like vvvvvv who think it should have been a finalist and are surprised it wasn’t.

@paul
“it’s about large numbers of people expecting a results that at least marginally reflect the community’s tastes”

the community can’t really have an accurate taste when half the games aren’t out yet.

I’d say the minimum number of games a judge needs to play is 30, then write up what they like dislike about them in some kind of judges comments system, then go back after playing them all and play them in quick succession (say 10 mins per game) then rank them in order from most liked to most disliked (playing quick games to re assess the order of two games). this list gets sent in, 15 of them will have been covered by another person as well, and the other 15 by somebody else.

if they can’t manage 30 games in 6 weekends (2.5 per day if you only analysed them at weekends) then why would they be considered as a judge.

because each judge analyses 15 of another judges games then you only need 20 judges minimum (2 reviews per game), however you’ll probably want 40 judges.

another thing you should encourage people who are playing the same game to share their opinions on it, so that if one doesn’t like a game the other may point out something about the game they hadn’t realized. and you should also encourage people to play more than the minimum number of games.

@robolee: There were more than 150 judges this year. With the current arrangement, each game gets played by more judges than you’re advocating for, for longer. The judges are required to leave comments. Also, there is a sophisticated judging backend: judges can and do leave comments for other judges. There was quite a bit of awesome, lively discussion in that channel this year.

@DanMacDonald: I’d be interested in seeing a survey with lots of participants: “Top games that didn’t make IGF finalist but should have.” There are always a couple really wonderful games that don’t make finalist.

well done everyone who made it! great to see Joe Danger and SMB on the list…

First off – super happy to see how much the judging process has improved. HUGE strides from previous years. It was especially nice to see that judges couldn’t pick their own games.

Personally not a fan of some of the picks, but can’t really complain just because I don’t like the taste of the judges.

@Paul Eres: I think you’re being a little unrealistic. There are now 300+ entries… assuming judges put in an hour per game (which imo is about the minimum it takes to properly judge a game, although after talking to some entrants I know who tracked IGF judge playtime, most judges only play about 15 minutes :/)… you’re talking 300 hours of time. That’s a full time job for 8 weeks. Definitely not feasible. It would be nice to find a way to thin down the herd a little so judges got to see a bigger sampling of games, though.

@Jeremy: Well, Star Guard is pre-8bit, does that count?

I wouldn’t even try to send my game to IGF if only a few judges test my game 15 minutes…

By the way 1/3 of this list is disrespectful for the rest of the 300 entries.

Participant selection should be better.

My suggestion for improving the judging process is to instruct the judges to evaluate each category for what it is, and try their best to follow the guidelines set forth in the competition rules for each category. The reason I say this is because this year I really felt that in a lot of cases, the judges might have just liked a game as a whole and then given it universally high marks… even though in individual cateogies I really felt that the games did not capture the spirit of the category’s guidelines.

I’m going to again point out the audio example with comparison to years past. The selection process for the audio category is as follows: “Scores will be based on the innovation, quality, and impressiveness of each Entered Game’s music and sound effects.” I can honestly say that most of this year’s audio finalists had audio that was good and fitting, but was not innovative. I could not in good faith say that the music/sound effects were nearly as impressive as other entrants or finalists from the past. Some barely featured any music at all. In the past we had finalists that attempted to do something unique and innovative with the audio aspect of the game; or we had phenomenal music and SFX. I cannot say the same for the finalists as a whole this year. It really just looks as though judges enjoyed certain games, and gave them universally high scores as as result without evaluating individual categories… and this is disappointing.

What has to happen is that the guidelines written in the IGF rules need to be changed, or the judges need to start evaluating each category specifically. As it stands, this is very disheartening to people who built their entire game around music and sound and really put effort into trying to do something innovative.

VVVVV is fun but it didn’t really blow my socks off. I’m actually really shocked so many people are complaining about it’s absence.

“This is one place where the IGF’s system is fine, they just can’t defeat the stupidity of the judges who don’t read the rules. :(”

@ShawnF: That is very true, it’s such a pity. My friends and I were eagerly anticipating the results of the audio finalists because we’re all musicians and had been following several of the audio-centric entries… it was very disappointing because it really did seem like the judges didn’t follow their own criterion for evaluating audio. The most obvious criteria not met by most of this year’s audio finalists I think was innovation (especially compared to years past). What’s the purpose of even having an audio category if it just ends up being a catchall for games the judges like as a whole?

I don’t think the finalists need to be specifically audio games, but the audio should at least stand out and meet the listed criterion. The judges should at least be issuing scores based on the “innovation, quality, and impressiveness of each Entered Game’s music and sound effects.” There were so many games this year where you could see that music and sound was a major aspect of game development. I can only ask that in the future, the organizers faciliate more emphasis being placed on the audio category. I’m sure it’s possible; after all, I felt that audio was accurately judged in years past… there were many innovative and stand-out audio games that were recognized in other IGF competitions.

I play both Enviro Bear 2000 (Nuovo) and Star Guard (Design).

I don’t understand…
I thought my game was a little better than that… I work so much over the year on that game. Write my very own engine, try my best at design and with graphics, pay a lot of money for good musics, create sfx myself and all.

Not even an email to announce me that I failed pathetically. What a shame for me to show to my family what won a finalist place over me while it is just what I hope for my game.

I strayed myself.

Mileage may vary – the feedback I got was really helpful. Things I learned from it:

* The game was submitted way too early in the production process. The judges were looking for something complete.

* Doing a special “everything unlocked” version for IGF was a really bad idea, because it caused the game to make next to no sense when played.

* The walking animations look wrong to some people.

* The character dialogue distracts from rather than supports the gameplay.

And comments about the game which I did feel were questionable were balanced out by the multiple perspectives, so hardly a cause for alarm.