The plying game: An inside look at the voracious, insatiable world of App Store reviews

Reviews are the lifeblood of online sales—as buyers, we rely heavily on the opinions of others to help inform our decisions. Sure, we research potential purchases first, but ultimately the one- to five-star ratings beside the things we want to buy are often the final arbiters over whether we add something to our carts or move along.

When it comes to the App Store, that rudimentary rating system is taken to the utmost degree. Our window to buy is often measured in seconds, and the majority of our purchases are based on impulse rather than need. Even if you’re looking for a specific kind of game or utility, the star rating holds a disproportionate amount of weight, especially when the app you’re considering is sandwiched between a dozen search results. Few of us do a much comparison shopping before we settle on which “Get” button to tap, so we place a great deal of trust in the cumulative score reflecting prior users’ level of satisfaction.

It’s the ultimate equalizer: The tiny stars are designed to tell you whether an app is worth downloading—whether it’s a free app or not—and they can influence us far more than even a five-mouse Macworld write-up (though those work pretty well, too). With some 1.5 million apps to choose from and a less-than-perfect discovery engine, a single star can often mean the difference between a hit and a flop. Unfortunately, those reviews aren’t always what they seem. 

Pump up the volume 

On Amazon, users don’t even have to have buy a product to rate it, but in the App Store, Apple ensures reviews are written by actual users—you can only leave a review after you’ve installed the app. This tight correlation between sales and ratings make it difficult for developers to track any direct influence ratings may have, and it’s unclear whether they have any effect on search results or top charts algorithms whatsoever.

Back in 2013, mobile marketer Fiksu surmised that ratings were finally being incorporated into Apple’s top-secret formula, but the company has since backtracked on that claim, dismissing it as little more than a blip on the radar.

“In our research, App Store rankings are still almost exclusively based on volume and velocity of downloads,” said Jeremy Sacco, Fiksu’s director of content and communications. 

For example, the SNL 40th anniversary app quickly broke the top 10 free apps just days after its release despite a mediocre 3.5-star rating, and Facebook Messenger has held the No. 1 position for some time with just three stars. Similarly, Snapchat’s 2.5-star rating hasn’t kept it out of the top 10.

“The data doesn’t show anymore that reviews directly impact your App Store chart ranking,” said Chris Shuptrine, senior director of product marketing at Fiksu. “That said, there’s certainly a relationship between reviews and the likelihood that somebody will be interested in your product and the likelihood that someone will download it.”

The rate debate

While all of the developers and marketers I spoke with saw value in the review system, there was little agreement on how much of an impact it has on visibility in the App Store. For example, Douglas Buchanan of Efecto, creator of Convertible (4.5-star overall rating) sees overall downloads as “the biggest factor affecting an app’s ranking.” But Max Scrobov, project manager for Yum-Yum Recipes (4.5-star overall rating), insists that his app’s high ratings has positively influenced its position in search results. And Readdle marketing director Denys Zhadanov thinks reviews do play a role in top charts and positions, “but not to a great extent.” 

Even Nick Lockwood, whose open-source iRate library on Github gives developers an easy way to offer review prompts within their apps, isn’t completely sold on how much they affect the standings. Upon development of the pop-up prompt, his app Rainbow Blocks (4.5 star overall rating) saw an immediate increase from single-digit reviews to hundreds, which translated into a “noticeable” sales boost. But still, he doesn’t think developers should put all their eggs in the review basket.

“I think ratings are necessary, but not sufficient, to ensure success,” he said, “The most important thing is to be high enough in the charts to be seen, and I don’t think anyone knows exactly what weighting is applied to review scores when ranking apps, but it certainly seems to be a factor.”

Pop-up culture 

Even if there might not be a measurable benefit, there most certainly is a psychological one. Snapchat may be able to overcome its poor rating on the strength of its brand, but smaller companies need all the help they can get, and a healthy set of stars makes their app that much more attractive to App Store shoppers.

That’s why developers crave these ratings and obsess over them, with many resorting to begging and bribery just to increase their score. The never-ending nuisance of pop-up ratings grabs are perhaps the biggest thorns in our iPhones’ sides—and you’ll find them all throughout your favorite apps, whether you’re clearing your inbox in Mailbox (4.5-star overall rating) or you just finished a particularly challenging level of Angry Birds Transformers (4.5-star overall rating).

The popular practice of pop-ups reached something of a fevered pitch in December 2013 when John Gruber’s influential Daring Fireball blog called out iRate as a major part of the problem. Lockwood in turn tweaked the default settings of his tool, thus eliminating how the dialogue prompt automatically reset after each new version. But while the change toned things down a bit, Lockwood hasn’t seen much of an impact on “either on the popularity of the library or its effectiveness.”

Most of the developers I spoke to haven’t see much blowback from users put out by the pop-ups (no matter how they operate), and all of them who utilized the device said the benefits far outweighed the risks. Readdle uses them across all of their apps, and Zhadanov believes strongly in the practice—if it’s done right.

“That means showing the right message to the right person in unobtrusive way, [like] kindly asking for a favor,” Zhadanov said.

Intuit takes a similar approach with its family of financial apps, which includes the popular finance tracker, Mint (4-star overall rating). 

“We have found that pop-ups in app can be effective in soliciting reviews, but only when timed and delivered correctly,” said Colette Crosby, Intuit’s director of global enablement and mobile marketing. “We want our users to love our apps, and when we’re confident they feel this way, we’ll ask for them for a rating. We set limits on the number of times we display pop-up prompts to not interfere with the overall customer experience.” 

And if the positive reviews outweigh the negative ones, then it’s certainly worth it. Zhadanov estimates that positive ratings “can affect sales up to 60 percent, especially if it’s a paid app.” Similarly, Crosby finds that “high ranking creates a virtuous circle. High ranking translates into more downloads, which in turn contribute to high ranking, and so on. A combination of high ranking, rating and download volume will tend to draw the App Store’s attention to your app.”

Star power

It’s a never-ending cycle, and even apps with thousands of reviews are constantly looking for more. Trivia Crack has enjoyed a recent surge to the top of the rankings thanks in large part to a 4.5-star rating, but parent company Etermax understands how fleeting App Store popularity can be, especially since there’s a visible reset whenever an update is pushed through.

“Positive reviews are good influencers,” a company spokesperson said. “Reviews are always a good way to impress people, attract and seduce them to download the game.”

But no matter how much users enjoy a particular game or app, getting enough reviews—especially positive ones—without a strong user base is increasingly difficult for developers. 

“It’s incredibly rare that satisfied users leave positive reviews by their own initiative,” Scrobov said, “but angry users will leave comments without a pop-up for sure.”

Take Monument Valley. An instant success, the game maintained a near-flawless rating for months after its release. But that changed when the developers released the Forgotten Shores expansion pack for a $2 in-app purchase. They were quickly bombarded with a torrent of one-star reviews, which drove their rating down from nearly five stars to less than two. The reason? Two bucks was apparently too much.

But despite the momentary rating dip, Ken Wong, lead designer of Monument Valley, said he didn’t see a noticeable fluctuation in sales, search, or chart position. The game’s top-notch rating quickly returned, and it comfortably sits in the top 10 of the paid puzzle games chart.

“We’re actually not really sure what impact the App Store rating has on sales,” Wong said. “Monument Valley has always had a high rating (apart from that infamous dip on the Forgotten Shores release day). Our principle is to deliver a quality experience first, [and then] everything else will follow. We made a great game. That’s it, honestly. There’s no prompt to give Monument Valley a score in the app, and no reward for doing so.” 

Cost benefit

While Monument Valley has enjoyed tremendous success without asking anything of its legions of satisfied, loyal users, other developers need to resort to more creative strategies to boost their ratings. Section 3.10 of the App Store Review Guidelines explicitly states that “developers who attempt to manipulate or cheat the user reviews or chart ranking in the App Store with fake or paid reviews, or any other inappropriate methods will be removed from the iOS Developer Program,” but there’s something of a gray area between what’s accepted and what’s not. 

A Weibo image recently made the rounds showing a young Chinese woman in front of a panel of several dozen iPhones, mindlessly downloading and rating apps. Apparently, it only costs around $70,000 to crack the top 10 using this method, but that particular type of get-rich-quick scheme is almost certainly a one-way trip out of the App Store. However, there are numerous ways to drum up reviews while still staying within the parameters of Apple’s rules. 

Apps starting from the ground floor can turn to any number of marketing companies that promise to get their apps more eyeballs, downloads, and reviews. Trademob offers a program called SmartBoost designed to specifically propel an app into the upper echelon of the rankings, where users and—presumably—positive reviews will follow. But “purchased installs” only boost the app up the charts, said CEO Heiko Genzlinger. “Developers should even go a step further and encourage their users to provide ratings for the app via a call to action button or dedicated social campaign,” he said. 

Facebook offers mobile app install ads that can be geared at specific age groups, but it can be a struggle for fledgling apps to hit an affordable cost per install. Based on Scrobov’s experience with Yum-Yum Recipes, users are leery of trusting apps without ratings. 

“We didn’t start the promo campaign until we collected enough reviews,” he said. “The cost was too high, but after we got five ratings, cost per install dramatically dropped.”

Pay to play 

Other marketers focus solely on ratings as a tool to drive downloads. One such example, AppWinn, offers raffles for anyone who agrees to download and rate certain apps. Prizes range from gift cards to iPod touches and Samsung Galaxies, but founder and CEO Jonathan Martinez says that “all of the ratings and reviews are unbiased and represent real feedback from iOS users.” However, a scan of the featured apps showed a propensity for four- and five-star reviews, yet Martinez insists AppWinn is dedicated to upholding the integrity of the App Store.

“As iOS app enthusiasts ourselves, we want to make sure we are only improving the ecosystem,” he said. “We have found that our users are fair in reviewing and developers have, more often than not, come out with great feedback in improving their apps further.” 

And then there’s good old-fashioned direct contact. While Scrobov and Zhadanov both have had good results targeting heavy users of their apps with requests for ratings, I found one app that takes it a step further. Instead of simply asking for a review after using their app, Shoeboxed offered me $5 to write one. It’s unclear how users are selected for the benefit or whether you need to meet certain criteria to qualify, but a week or so after I downloaded the app, I received an email offering an Amazon gift card in exchange for writing a review. I was suitably skeptical, but upon leaving an honest 4-star review with a bit of criticism about upload speeds, sure enough, I got my reward. I’m not sure if this is a common practice used by the company—which declined to be interviewed for this article—but the receipt-tracking app is hardly starving for reviews. The current version (updated January 15) already has 149 reviews, averaging four stars. 

The developers I spoke to who were using marketing campaigns all aimed to keep costs under $1 per install, so offering $5 for a review of a free app (albeit one that offers a paid subscription plan) does seem excessive. But it shows the lengths smaller developers will go to, just to build up their profile. It’s a dog-eat-dog world in the App Store, and without a solid rating it’s easy to get lost and, eventually, swept away and forgotten. 

But still, the practice doesn’t sit well with Monument Valley’s Wong.

“I think annoying, pestering, and bribing players is a bad habit,” he said. “It’s a desperate attempt to compensate for lackluster, uninspiring, repetitive experiences. I’d much rather have a relationship with the players where we provide them with an experience that’s polished and special enough that they leave us a rating or review of their own will.”