Valve Software has amended its policy governing user reviews of products in the Steam games store. The reasoning behind the changes offers valuable insight into the potential pitfalls of user-generated reviews in general.
The Steam client app allows owners of any game (or other product sold via the service) to post a review. As with Amazon or other online stores, these reviews are visible on each product's main page, and are a major influence on potential customers who may be considering the product.
It's a given that user-written reviews will vary widely in credibility and usefulness. But it's a serious problem if the reviews process as a whole becomes subject to systematic ‘ballot box stuffing' by product creators. There have been famous cases where numerous favorable Amazon reviews were found to have been posted by employees of the manufacturer. Such a grab for short-term gain by one company comes at the expense of decreased confidence in the whole online retail ecosystem.
In the Steam games store, that sort of abuse had apparently reached disturbing proportions. Valve has attempted to identify and block the mechanisms that were allowing this to happen.
Positive Reviews
"An analysis of games across Steam" says the Steam blog, "shows that at least 160 titles have a substantially greater percentage of positive reviews by users that activated the product with a CD key, compared to customers that purchased the game directly on Steam." The posting admits that there are "legitimate reasons" why this could be true for any particular game. "But in many cases, the abuse is clear and obvious, such as duplicated and/or generated reviews in large batches, or reviews from accounts linked to the developer."
Steam automatically ensures that reviewers do own the game they're critiquing. But many users don't buy the game directly through Steam. Instead, they acquire Steam ‘unlock keys' from various other sources. One popular example is Kickstarter, where avid gamers are given Steam keys in exchange for backing a game's development. The numbers are not huge, but significant nonetheless. For instance, reviews of the recent game No Man's Sky, from Hello Games, come from 68,451 Steam Purchasers, vs 1,062 players who acquired the game via Key Activations.
The concern is that a developer can easily give out unlimited numbers of unlock codes, and thereby recruit an army of favorably biased reviewers. Again using the example of No Man's Sky, the proportion of outside purchasers might be small, but a block of a thousand positive reviews could still be very significant. (This is just as a random example, by the way. No allegations have been made about this particular game.)
Review Scores
Valve, always reluctant to make sweeping changes in the Steam ecosystem, has responded with a typically minimalist tweak. Gamers who obtained a game using an outside activation key can still post reviews, and shoppers can still see them. But these reviews' thumbs-up and thumbs-down ratings are now ignored when generating a game's overall score.
Those summary ratings appear at the top of the game's store page, and are the first thing a potential customer will see: "Overwhelmingly Positive," "Positive," "Mixed," "Mostly Negative," and so on. The more negative ratings appear in a nasty red color.
"To make it easier to tell whether customers overall would recommend purchasing the game, we created a review score," says the Steam blog. "We've intentionally kept this score as transparent as possible, by simply calculating the percentage of positive reviews."
"But the review score has also become a point of fixation for many developers," adds the blog post, "to the point where some developers are willing to employ deceptive tactics to generate a more positive review score. The majority of review score manipulation we're seeing by developers is through the process of giving out Steam keys to their game, which are then used to generate positive reviews. Some developers organize their own system using Steam keys on alternate accounts. Some organizations even offer paid services to write positive reviews."
According to Valve, its new approach to calculating the aggregate scores resulted in a change in score for about 14% of the games on Steam, some going up and some down. Many of the changes are minor, resulting from a score that happened to be on the borderline between two of the rating tags.
Access Filters
It's too soon to know how much this will benefit the overall review system. But Valve continues to make adjustments. Most obviously, it has created a new and far more detailed dashboard that shoppers can use to access reviews. Now, users can set filters to view only reviews from Steam Purchasers, or only those from Key Activations owners. They can view only Positive (thumbs up) reviews, or only Negative (thumbs down) reviews.
Users can also sort out just the most recent reviews, to see how the opinion of a game may be evolving over time. Or they can select to see the reviews that have been marked as Helpful by other viewers.
That latter attribute is causing Valve the most soul-searching. "There are some titles where the most helpful reviews don't seem to accurately match the general customer sentiment," says the Steam Blog. "For example, there are a couple of prominent titles that have review scores of ‘positive' but all the reviews marked as helpful are negative. We need to look at this to figure out how to represent cases where the community has highly divergent opinions."
"There are some titles," notes the blog, "where a small group of users are able to consistently mark specific reviews as helpful, and as a result can present a skewed perception of what customers are saying about the game. This is obviously not ideal, so we're looking at ways to ensure that a few users don't have outsized influence over the system."
Transparent Approach
The success of online vendors like Valve and Amazon is undeniable. But even those goliaths are still learning about the nature of their interaction with customers.
Valve, in particular, is unique in its level of interaction with its user community. Even minor changes in the Steam store tend to generate vast controversy among avid users. But Valve's logic is always sound, and guided by shrewd analysis of data from over 130 million active users.
The transparency of Valve's current approach to user reviews is thus a great chance for all of us to increase our understanding of how these community-driven systems work -- how they can be tuned to give more-useful results, to increase customer confidence, and to produce a more frictionless buying experience.