Hi there! My name is DDRKirby(ISQ) — I’ve been participating in Ludum Dare enthusiastically for the past 5 years, having submitted 12 games to both the compo and jam events. 6 of these entries have placed in the top 10 overall (not just trying to boast — this will be relevant in a later part of this post).
As many of you know already, there has been a bit of a situation regarding the upcoming August LD event. A poll was made, with the end result being the planning of an August LD event on the current LD website without using LD’s categorical 5-star rating system. As you might expect, this also triggered some community backlash.
I am not affiliated with Ludum Dare officially and am in no position to make demands on how the August LD event should or should not be run. However, I wanted to provide a personal viewpoint on some of these issues.
My goal in writing this post is two-fold:
To highlight some of the major disadvantages and problems with LD’s rating system.
To offer some suggestions for how we can best help the no-ratings August LD event to succeed.
Please note that I am NOT attempting to convince anyone that Ludum Dare should not have a rating system. I think that most people understand that having a scoring system can be extremely useful and valuable. However, I think it’s important that everyone also understand the potential issues and problems involved with the current rating system and why many of us would prefer to have an August LD without it. Let’s get right into it:
Winning Ludum Dare is a Big Deal
One of the things I really like about Ludum Dare is the fact that there are no monetary prizes. As the rules page puts it, “Your prize is your product.” (and to a lesser extent, the feedback that you receive on your game as well) This is supposed to help foster a low pressure, non-competitive environment where people can simply focus on making cool games, which is great!
However, one thing I’d like to make clear is that winning Ludum Dare is a Big Deal. Being highlighted as one of the winners, especially in the Overall category, guarantees you and your game a TON of exposure, not only to fellow Ludum Dare entrants, but also by gaming portals, indie game sites, YouTube videos, twitch streams, and more.
When Ripple Runner won 2nd place overall in LD29, I got a huge spike in followers and network traffic. Not only did several gaming news sites feature it in articles, but web portals started outright STEALING my swf file and hosting it themselves to profit via ad revenue.* A Korean stream video of my game later went viral, hitting over 80,000 views.
I sell my game soundtracks on my bandcamp site. Despite having a minimum price of $0, sales from the Ripple Runner soundtrack earned me $200 in profit after LD results were announced because of the wide attention garnered from my 2nd place ranking and would go on to earn me over $350 in total in the following months.
In other words, placing 2nd in Ludum Dare had MONETARY implications for me, almost as if I had received a $200 cash prize — and this is for a pay-what-you-want album! I can only imagine that things would have been even bigger if I had come in 1st place, or if I had charged $5 minimum per album sale.
I would love it if Ludum Dare were not about winning and losing, but the fact is that the benefits gained from placing highly are very real and from this point on I have always felt mental pressures in the back of my mind to do my best to score highly as a result.
This unfortunately provides a large incentive for cheating the system.
Ways to Cheat the System
PoV has already outlined in his post that there has been evidence of cheating within LD, including:
A user that created multiple accounts just to give their game 5 star ratings.
Games that have more votes than downloads (i.e. people rated them without even playing).
Users that voted for many entries in an extremely short period of time (again, indicating fake “throwaway” votes that were made without playing the actual game).
These things are not unique to Ludum Dare. This post from GameJolt outlines some additional ways in which people have tried to cheat around the rating system in their own game jam. Remember, these are community game jams without monetary prizes.
An individual who hacked into/used multiple Facebook accounts to sign up and make it appear as though different people were rating their game.
People giving their own entries many “4-star” ratings to avoid arousing suspicion from giving themselves 5-star ratings.
Around 60 entries were found to have cheated by either starting early, or by using disallowed 3rd-party assets.
If I were 100% serious about winning a Ludum Dare compo and was prepared to cheat the system in order to do so, I would probably leverage some or all of the following additional strategies:
Enlist one or more artists to draw artwork and animation for the entire game, but still submit it to the compo event.
Use tilesets, background animations, and menu systems made completely ahead of time.
Subvert the time limit — when submission hour ends, submit a broken link to the game, but keep working for another 48 hours. When I’m done, upload an incredibly polished game and claim to “fix the broken link”.
Create many user accounts using different IP addresses. Submit random scrappy game projects under those accounts to avoid being flagged as fake accounts. Using these accounts I could upvote myself, then be on the lookout to downvote other people’s games that seem popular. Other randomized votes could be applied to avoid easy detection.
Some of these methods would be very difficult, if not impossible, to track using the current LD rating system. Of course, any of them would constitute blatant cheating and disregard for the rules and as such I would never do any of them. But the point I’m trying to make is that there is both incentive and capability for cheating here. In addition, you wouldn’t even have to do very much of it: For Ludum Dare 28, our Jam entry missed hitting 1st place by a mere 0.02 points (4.39 vs 4.41). As few as THREE fake 5-star ratings would have changed the outcome.
There are a multitude of ways in which the system could possibly be augmented and improved in order to prevent, mitigate, or detect the effects of cheating. Again, the gamejolt jam post illustrates one example of what this looks like (note that it involves coordinated efforts as well as manual intervention). However, it is IMHO unrealistic to expect any of these changes to be implemented for the upcoming August LD.** How this should be dealt with in the future is outside the scope of this post. I merely wanted to illustrate the potential problems that exist in the current system.
Ratings Can Cause Frustration For Jammers
So far the cheating issues I’ve described perhaps only matter significantly for top-ranking games, where there is a lot of potential recognition and attention on the line. But for anyone, having your LD game judged on a quantitative — yet extremely subjective — numeric scale can be frustrating in other ways as well. This post and this post highlight some of these frustrations, including the following:
Numerical ratings encourage direct competition with and comparison to other games. Instead of supporting each other, we can become frustrated when “bad” games score higher than ours and are incentivized to rate other games more harshly.
Focusing on quantitative scoring results in a larger emphasis on evaluating a game as great or poor based on a set of numbers rather than providing constructive feedback on what could be improved. (What’s more important, a few well-written reviews or a single aggregate number?)
Some users have complained that taking a straight average of scores means that games with less ratings tend to have better chances at scoring particularly highly due to how variance works.
“Celebrity” competitors sometimes*** being able to leverage their past success and popularity to earn higher scores (probably without even meaning to).
“Gimmick” entries being voted consistently higher in certain aspects (e.g. rhythm games or audio-centric games regularly voted #1 for Audio).
Wide voter preferences for certain stylistic choices (e.g. pixel art, chiptunes).
Ratings that just don’t make sense or indicate that users to not understand how to rate games properly (e.g. Games with NO MUSIC OR SOUND receiving audio ratings between 1.5-2 stars).
Ratings that are based off of the post-compo version of a game, not the 48 or 72 hour compo version (you can tell by reading their feedback comments).
Cheating too easy (as described above in the previous section).
I am NOT trying to say that these factors are pervasive and that our rating system is inherently terrible. These are just some potential disadvantages that I think people should be aware of and paying attention to.
Personally, I believe that the pros of a rating system can well outweigh the cons if handled properly. However, I’m also interested and hopeful to see what a Ludum Dare without ratings will be like, if only as a test. If it fails horrifically, we’ll know to never do it again in the future, but I’m hoping that we can prevent that and instead make this August LD a success by focusing on qualitative feedback and community interaction instead numerical ratings.
This is where you come in.
How You Can Help – Leave Detailed Comment Feedback
One of the main benefits of the categorical rating system is that it can provide you with a detailed breakdown of what areas you did well in and what areas you need to improve on. This is extremely useful, but could we do the same thing by using comments?
Here’s an example of a comment that I would give in a typical Ludum Dare, alongside 1-5 star ratings for each category:
Cool game, and nice job on your first LD! I especially liked the animation of the main character. Next time try to add some sounds!
For our August LD, we will not be using the categorical rating system. Therefore, I will be using the following categorical comment template to provide more detailed feedback:
Innovation: This was not too much different from a standard platformer here, but I didn’t really mind.
Fun: The game was pretty fun at the beginning, but got a bit repetitive, especially the water level which I think involved too many difficult jumps. Maybe adding another powerup would keep the game interesting all the way through?
Theme: The main character was a goat! This definitely fits.
Graphics: I really liked the animation of the main character. The background for the fire level was a bit too distracting, though.
Audio: The music was a bit repetitive, but catchy. You should try to make some sound effects next time, with a tool like sfxr, bfxr, or Labchirp — it only takes a few minutes and goes a long way towards making your game feel more complete!
Humor: N/A
Mood: I really liked how each level had a different theme. I think it worked very well to establish a mood. As I mentioned, sound effects would really help out here!
Overall: This was honestly one of the better games that I’ve played so far this LD. It was a little buggy and could definitely still use some work, but I would love to see you work on a post-compo version. Feel free to leave me a comment on my game’s page if you do!
Congratulations on finishing your first Ludum Dare! I’d really appreciate it if you could take the time to leave some feedback on my game in return. You can find it at […].
IMHO, this type of detailed feedback is infinitely more valuable than a simple numeric rating. I would highly encourage others to provide comment feedback in a similar style.
How You Can Help – Comment Back
Several people have expressed concern about the ability to receive enough feedback and comments on their games due to the lack of ratings and rankings as voting incentives.****
You’ll notice that at the end of my example comment I requested the game’s author to provide me with feedback in return. This sort of “I’ll rate yours if you rate mine” trade has been a great way to get additional feedback in past LD events, independent of the “coolness” system.
For this Ludum Dare I will be making it a point to return the favor by playing and giving feedback to anyone who leaves feedback for my game. It would be great if others could do the same.
In addition, we should continue to leverage the compo site itself to help out jammers in need of additional comments. Don’t have enough feedback? Post some interesting screenshots of your game and ask for comments. Not sure what games you should play? Make a post inviting people to comment with links to their games. Let’s all help each other out! 
How You Can Help – Curate Outstanding Entries
Another benefit of the rating system is the ability to select the cream of the crop based on the ratings accumulated through the voting period. The best of the best should absolutely be selected and called out so that we can all see what truly amazing games can come out of a single weekend.
For the August LD, we won’t have any sorted ranking lists from which to select these entries from. Therefore, the onus falls on us, the jammers, to informally curate the top entries.
Some LD jammers have already been doing this in the past by posting lists of their favorite entries, or even creating video compilations or twitch streams of notable entries. This is great, and I would encourage more jammers to start making these sorts of posts, and/or upvoting other posts if you particularly liked a game or two. I have personally never made a “My personal favorites of LDxx” post on this site before, but I plan to for LD36. This is your chance to shine the spotlight on whoever you think deserves it most.
I know that this August LD has been on shaky footing and I understand that unfortunately, not everyone will be entirely happy with its outcome. I really hope that the community can come together and give it their all despite that. If we put in a little extra effort, we can definitely ensure that Ludum Dare 36 will be a great success.
(It would be awesome if some of these recommendations could be highlighted in a keynote or an announcement if people agree that they are good ideas. However, I must emphasize that they are only my personal suggestions and again, I am in no position to tell other people how they should or shouldn’t handle LD.)
Thank you so much for reading. See you all in a month, and I’m looking forward to playing (and commenting on) your entries! <img src="https://s.w.org/images/core/emoji/72x72/1f600.png" alt="