The Art Galleries of SL List is available at http://sasun.info/ArtGalleriesOfSL.htm.

New visitors, please read: About This Blog

Monday, May 16, 2011

Gallery Rating System revamp

I have been doing some thinking on the current gallery rating "5 star" system and I am seriously thinking of doing a complete overhaul some time soon.

I would like to dispense with the star ratings and instead have a simple Like model, similar to Facebook. The star rating graphic on the kiosks and tour HUDs will be replaced with a simple "Like" button and if people like your gallery, they can click it.

One advantage is that it's very simple and will match the same model that LL uses on the Destination guide.

I will convert all 4 and 5 star ratings to a "Like" in the new system when I make the change and simply delete all 1, 2, and 3 star ratings. On the gallery list page (which is also going through a complete redesign btw) all you will see is the  number of Likes, just like facebook does. There will no longer be 1-5 star votes - only likes.

This means that a gallery that does a good job of getting people to Like them on their gallery kiosk will be high on the list, so it's more a simple numbers game than trying to get 5-star votes and no bad votes. The current system is just too problematic in that the vast majority of the votes are 5 star votes, so one 4 star vote can send someone tumbling far down the list. In effect, everyone is giving galleries either 5 stars or nothing already. The big difference in the sorting order is that the raw number of votes will now be very important, and in the current system it is much less important than the average star rating, which is what they are sorted on now. The current system uses a Bayesian rating system which "penalizes" galleries with a low number of ratings so that a single 5-star rating won't plop you at the top of the gallery list.

This will also completely remove any griefing done by people rating galleries 1 star for personal reasons, and there are, for reasons I can't comprehend, a few people that give out nothing but 1 star ratings. Which is just completely pointless and it pollutes the database with garbage. Why go visit galleries if you just hate them all and give them lousy ratings? So this will completely solve that problem too.

The new system is not perfect either, since galleries that are strategically placed in very high traffic areas will probably get more clicks than a gallery that's great but is out of the way. So both systems have their faults but I think the facebook Like model is going to work out better in the long run. I'm still gathering feedback on that idea, so it's not set in stone yet, but that's my thinking. I'd love to hear what you think!


  1. I think it's a good idea. A rating from 1 - 5 is very subjective. Just like or not is better I think and the more 'likes' indicate the rating best. Wouter Collas (InSL)

  2. I think it is a really good idea to simplify this system, and it will bring gallery ratings more in line with what social media does today, so the learning curve should be minimal.

  3. There are some visitors who will click the "Like" button every time they see one. There are others who will not do so unless they are completely astounded. Even so, it makes it an easier decision than having the visitor mull over "how many stars". Approval is active, disapproval is passive. Maybe once a gallery reaches a certain level of notice, say is gets 100 "likes", you could send over a professional reviewer to examine the venue, and write a Michlen Guide style blurb. Second Life could use a decent Michlen Guide. Just like the RL Michlen Guide, a small fee per copy would be reasonable. No venue owner is allowed to solicit a mention in the Guide, and no venue owner is allowed to know when or by whom they are reviewed.

  4. Your reasoning for the proposed changes, is perfect. I get mostly 'nothing' but then, I don't promote as well as I should. (note to self: Remind people that the kiosk is there!) A like button would remove anyone griefing for their own reasons with one star. Very nice, thanks for sharing.

    Treasure Ballinger
    Cape Able Art Gallery
    Cape Able, a Virtual Ability Inc, owned sim

  5. I have mixed feelings about this approach, Sasun. Frankly, I think the popular rating of galleries is a disservice and impediment to entrepreneurship.

    There are many problems with straight up or down votes. It really only works with something ephemeral. Otherwise the one with the most votes will always be at the top. A new gallery will have to exceed the total votes of an established one by a strong margin over time just to catch up eventually. To keep it fair and competitive perhaps it could take the average Likes over the most recent 3 weeks or whatever. Thus a really popular gallery would have to remain popular to keep its standing and would moderate peaks from major events. The down side is that people would have to keep voting for their favorites in order to keep them on the list, but at least it would be a dynamic process.

    Up or down voting lacks nuance. You're no longer rating the best galleries, only the most popular. You are changing the question from "How much do I like this gallery?" to "Do I like this gallery?" If I like it at all, it gets the same vote, even if I only like it a little compared to the one next door that I liked a lot. I'm not sure that's useful.

    I think transferring the top star-scaled ratings to Likes is not fair. It's not the question that was asked when the vote was cast. The rating systems are not equivalent because they do not represent the opinions of the voters in the same manner.

    I can appreciate the problems and complexity of having a rating system, (I'm dealing with them myself at UWA) but I would argue that eliminating ratings altogether is better than having a simple popularity contest that can be manipulated by campaigning and provides no useful information on which to base a decision to visit a gallery.

  6. Sasun, I think it is a brilliant idea. I think sometimes people give a gallery a poor rating because they arent clear on which star to click on. And ratings griefers? ugh - that never occurred to me :((

  7. Sasun, i think its a great idea. You have my support if you need to pilote or test a system. Cheers Maded Shepherd

  8. Hi Sasun, the basic idea seems fine, but aside from galleries in high-traffic areas, it disadvantages new galleries, no? Maybe the number of Likes should be divided by the number of months the gallery has existed in order to level the field. Also, will you send out new kiosks to work with the new system, or just convert the stars people send into Likes?

  9. Hm yes that's something I didn't consider. Long-running galleries may have accumulated a large numbert of Likes. I really like the idea of Likes/month, I'll think about that. There will be new kiosks going out since I will need to totally redo the Likes button, but yes in the mean time any vote of 3 stars or below will simply be ignored and any 4 or 5 star vote will be converted into a "Like", that's what I'm thinking.

  10. @FreeWee, good comments. I hate to throw out all the previous votes and start all over, but that is something to consider. I agree with what you are saying about the question being different, but here are the numbers:
    5 stars: 82%
    4 stars: 8%
    3 stars: 4%
    2 stars: 2%
    1 star: 3%

    So in effect everyone is basically doing almost the same thing as clicking "Like", with a small number of 4 star votes and on down. At this point it may as well be "Like" or no vote and in effect penalizes the galleries with a huge number of ratings that a lot of people love. Looking at the vote distribution I think the current system just isn't really working out since most people just vote 5 stars if they like the gallery at all. I do agree that doing a time-based "Like" count is a very interesting idea.

  11. @Scire that is a very interesting idea. I've thought about getting together some type of "review commitee" of qualified reviewers that can go around and review some galleries. I like the idea of accumulating a certain number before a review is put in the "needed" queue, and we could have editor's picks and whatnot, like the Destination Guide, which would be interesting.

  12. I agree as well, Sasun. I like your thinking on this new approach. Thanks for opening this for discussion. Cheers -- Teo.

  13. Sasun, I think a simpler rating system is an excellent idea. I'm totally for the like button. I think people may get 1 star ratings now just because visitors do not understand the 'worth' of the rating and think one star is a good thing. One really has to explain how to rate, and some galleries do give out notecards for how to, but how many visitors take the effort of reading it? A simple like button will solve so many problems. And yes, there excist griefer rates as well, at least they won't get a chance in your new system. So I'm all for it.

    Not sure about the month dividing idea, I think it's another punishment in the system, in this case even for having your gallery up for some time already, so for being a loyal kiosk gallery, I don't think that's fair. The new galleries already have a feature of being listed as new on the website, and everyone who visited the older galleries can visit them too, and find them easy by clicking the 'new' button on the website list. I think it must be easy to catch up with ratings for new galleries this way, especially with a simple like button that people will be much more inclined to use. There's always in world groups to advertise for new galleries and remind visitors to click the button if they like the gallery. I have old galleries, who move around, and regularly new galleries too, so in the new system I'll just have to work harder for the new galleries to get the visitors hit the like button if I want them more high up the list. I think that's fair, and the older galleries high up the list have the right to be there if so many people have liked it.

    I wonder what happens with the reviews in the new system. It's hard to get people to write a review at this moment. Maybe a 'write a review' button may make that easier (next to the like button) but then again, griefers may use it for same reasons they do now. Not sure if linking it to the like button will help (like you can review only when you have rated, like it is now), maybe call it 'tell us why you like this gallery' instead of review, I don't know. There'll always be jerks out there, unfortunately.

  14. Thank you for allowing anonymous posts, it gives me the freedom to speak candidly...

    You have good data but you are reading it wrong. Yes, 82% give 5 stars. But it is actually very rare for a random visitor to give 5 stars when this type of system is used. What you are seeing here is that about 80% of the ratings you are getting are from people with some personal interest or enticement. There are some people at the top of the list with just a few pieces of low quality art. They have tons of 5 star ratings because they are constantly throwing parties and events. Not because their gallery is any good. I saw one gallery that had just opened, had no art at all anywhere in the region, had door spam saying they were looking for artists, and had 10 ratings at 5 stars.

    Once you realize this you will understand that what you are proposing is to throw out the actual legitimate votes and only keep the highly questionable ones. What you should be looking at is the number of clearly legitimate votes by random visitors, the 1-4 stars range. The real problem is that there are not enough of these REAL votes to counter the enticed votes. It looks like only 15 to 20% of your votes are truly candid. And, quite frankly, the reason is that the list is not intuitive for the casual visitor. It is only getting use from those with a vested interest. Honestly, before I got personally involved with the art scene in here I tried using your kiosks, HUD, and site several times and each time I quickly abandoned it as useless. Both because of poor design and poor results. I have talked to a number of others about it and they all relate the same experience. In fact, I have tracked the landings and the only visitor I have had come in at the kiosk landing point was you, Sasun, shortly after we set it up.

    Being on this list is only good if the list is actually useful to the casual visitor. So my primary objective here and in my own ratings is to try to improve the list. If a true art lover visits the first three galleries on the list and they are junk, they are going to turn to other methods of searching for galleries (which is what I did). And just to say it straight out, all the ratings of my own gallery have come from my friends accept 1. All my ratings are 5 stars accept that one. And that single lower rating came from another gallery owner on the list. And that is out of nearly 1000 visitors to my gallery. That pretty much tells the whole story right there. No one else is using your kiosks and that is the root problem. Changing the rating system won't fix that.

  15. I think you make some good points, but I want to debate a few of them. I don't think it's true that random visitors do NOT give excessive 5 star ratings. If I take all the galleries (607) and remove every gallery that has 10 or less ratings to account for what you say is "friends and family" ratings (and there is some truth to that, yes!), which leaves 217 galleries (about 1/3 of them) and look at the average rating, it's still a very high 4.6. Even if we look at the galleries with > 100 ratings (5 of them) the average is almost exactly the same - 4.6. There is no way those galleries have > 100 friends and family voting them up! The vast majority of those votes simply must be random visitors. No matter how you slice and dice it, the average is just very, very high and is the same.

    There are always going to be some exceptions and bad apples. I have visited all the "top" galleries and they do not have "a few pieces of low-quality art". Yes there may be an exception and if there is one of them way up on the list that doesn't have easily discoverable art there (note that sometimes the landing point forces you far away from the actual gallery) I'd sure like to know because that is highly unusual that a top gallery has little to no art in it. That smells like something fishy and I'd definitely like to investigate. Fraud happens and I do regualr reviews to find them (more on that later!). As far as galleries having a bunch of nothing but 5-star ratings, yes that is true of a huge number of galleries, and my hope is that the list is self-correcting - if something suddenly bubbles up to the top, I hope that people go there and RATE the gallery - if it's nothing special, then a few honest ratings will move it way down the list. I'm not saying go ding some new gallery, but if people are rating them what they really feel it is worth, the ratings will sort themselves out. One of the best things you can possibly do if you find what you feel is a 3-start gallery that's rated 5-stars is to rate it 3 stars. It may not make much of a difference in their rating number, but it will definitely immediately shift the rankings.

    Galleries that throw parties and events and encourage people to rate the gallery via the kiosks are doing the smart thing. If you open a gallery in an obscure, low-traffic or hard to find area, and never promote it, you will never have any visitors and hence little to no ratings (nor sales!) regardless of how great your work is. Galleries with low numbers of ratings are penalized by the Bayesian ratings system and pulled down the rankings, so it is a very good thing to get more ratings - you SHOULD host events, expose your art, get more people interested in SL art, and get more people to rate your gallery. I've even made special ratings kiosks that you can strategically place around your gallery to encourage more ratings. If your kiosk is hard tdo see and you are not using the special ratings kiosks, well now you know what to do to get more ratings. Does your visitor counter encourage people to rate the gallery? That would help too. The galleries that are smart and do these things to get more ratings are the winners, and deservedly so. But raw number of ratings will only stop you from being penalized for having low number of ratings - after that, it's people's opinion of you, your gallery, your work, and whatever else is influencing their rating at the time, which I admit can be impossible to predict.

    One other thing that I have done recently is that I realize that it's not clear enough that you can click the little stars on the kiosks. I've changed the text on my latest release of the gallery kiosks - it now says "CLICK here to rate this location" so hopefully that will be easier to understand what the call to action is and encourage more ratings. If anyone wants a new kiosk with the improved wording, let me know!

  16. Also when you say the list is not intuitive for the casual user, I'm not sure whether you mean the online listing of galleries that I just redesigned, or what... do you mean it's not intuitive *how* to vote on the kiosks? I'd have to agree with that and welcome any ideas how to encourage more gallery owners to get visitors to rate via the kiosks or the tour HUD.

    As far as being useless, the numbers I am tracking tell a different story. Please take a look at http://www.sasun.info/clickerReport.aspx?type=SLURLclick&source=gallery. That chart shows the sum of 1) clicks on SLURLs on the website to that gallery and 2) "views" of that gallery on the tour HUD (there is no way to know whether they actually teleported after a gallery comes up on the HUD map). So those numbers can be taken as "page views" or "eyeballs" in web parlance - it's not a guaranteed avatar landing at your parcel, but it's an avatar looking at your gallery info and a visible teleport button to go there. I'd guess that a large percentage of people that get as far as an actual map popup to your gallery actually click it and go there, but I don't know what that is. So that's free advertising and promotion right down to an actual map popup in SL to that location that you don't have to pay for. There are 15,169 "map views" of this sort for existing galleries (views for galleries that no longer exist are not shown but there are a LOT over the years).

    I'd also like to find out what you mean by poor design - the web listing design, the kiosk design, the rating system design, which is it?

    As far as poor results, if you are expecting a ton of people to show up at your gallery via the kiosk simply by rezzing a kiosk, it's not gonna happen. With a low number of ratings your gallery will forever be stuck near the bottom of the list which no one will ever see. Until you get a nice pile of ratings it just won't get very high on the list and even then, as you can see from the previous chart the top galleries really get a lot of eyeballs and it falls off pretty rapidly after that.

    It's super hard to look at the list and say, it's not accurate, because everyone's opinion is completely different. I definitey do not see any junk near the top of the list, so from a gallery visitor's perspective, this is way better than random chance or the destination guide as far as finding a good gallery to visit. You can read the text reviews and you can sort through them and read them to decide what to visit. I'm sure anyone reading a well-written intelligent two sentence review will have more faith in that over some misspelled two word crappy review.

    All that being said, this system is definitely not the best it can be and is in need of some kind of revamp or improvement, which was the whole purpose of this article to begin with :) I'm still giving this whole thing a lot of thought. There seem to be just as many problems with the Facebook "Like" model (which the Destination guide uses) as there are with the existing system so it has not been changed yet. As far as what would be better I am still toying with the idea of having "verified" raters - either people that rate a lot of galleries (or somehow discounting the large number of "drive by" raters that have only rated one gallery, ever) or perhaps designating certain verified real art critics as having a more influential vote - like a "x10" bonus for every one of their ratings. Or perhaps a different view of the galleries using different ratings techniques (like ignore all 1-vote ratings or something). Still pondering all this.

  17. Hi Sasun.

    Thank you for the thoughtful reply, but your logic is a bit fuzzy yet. For example, 'If I take all the galleries (607) and remove every gallery that has 10 or less ratings to account for what you say is "friends and family"' Sorry, but that does not account for it at all. I could easily get 200 friends in SL to give me a 5 star rating without even looking at my gallery. The reason I don't is that it is not an effective way to advertize. It would take time and effort better spent in other directions (and I would lose some friends in the process). But many think it is effective.

    You might find this interesting - http://www.lifewithalacrity.com/2006/08/using_5star_rat.html

    They mention in a round about way that, if the people come to the rating system through a show, they give the show overly high ratings. If they come to the show through the rating system, the ratings are more thoughtful and meaningful. This is what I have been saying and I will get back to that in a bit.

    Your click through data is interesting (i'll give it the benefit of the doubt and call it click through instead of page view), but shows exactly what I said...no one is seriously using this list. Many days I get more visitors to my gallery than you get click throughs on the entire list. Some days you get as low as 2 click throughs on a list of over 600 galleries. You are consistently under 50 clicks per day and only went over 100 clicks on one day in the past two months...the day I and some friends spent the day visiting galleries on the list.

    If you adjust for the gallery owners, especially new ones, checking out the list, you essentially have zero click throughs. You say that, if you are way down the list, you are not likely to see any traffic. But, as you actually mentioned, if you are high on the list you are not likely to see any real traffic from it either. I'm not going to go to a whole lot of effort to fight over and average of about 30 clicks a day divided among 600 galleries with almost none of it coming from actual shoppers.

    I get plenty of traffic. The important thing is that most of it is targeted traffic that is actually shopping. This is one reason that after only a few months my gallery is already making a profit over expenses. I pay attention to what really brings in shoppers and throwing events is a total waste of time and effort. So far, this list is as well. The Kiosk is getting many times the exposure from my gallery as I am getting from the kiosk. The question is, why, with all the exposure this list gets from so many galleries, is it not being used?

  18. You made my point nicely again when you said that people who have events and drag visitors to the Kiosk to rate them are the smart ones. Why is it smart to use other methods of promotion to get people to your gallery to help you with a method of promotion that doesn't even work? It is much smarter to focus your promotional efforts on methods that actually bring in paying customers or true art lovers, then leave them alone to browse while you focus on producing the kind of art they really want. The whole idea of the list is to get traffic to the galleries. You are saying the smart way is to get traffic to the gallery other ways and get them to the kiosk through that. If I get traffic to my gallery my next goal is to get them browsing, not rating on something that will do no good.

    If people were actually using the kiosk to find galleries they would be far more inclined to use it to rate, and to rate thoughtfully. Even with all these event-people rating, it doesn't get them using the list. I would bet you get far more ratings overall than you get click throughs. They are coming to the kiosk through the gallery...instead of coming to the gallery through the kiosk. Ask yourself where their natural allegiance will be in those two scenarios. That is what the studies of such ratings systems show.

    So your idea to eliminate anyone who has only rated one gallery has merit. Better yet, weight their rating based on the number of galleries they have rated. The more galleries a person rates the more likely they are to make good comparative ratings. I think you will find that those who rate a lot of galleries are far more likely to use all the different ratings instead of just 5 star or 1 star. Eliminating one-rating people is only a patch. What you really need to do is increase the number of multi-rating visitors and the only way to do that is to get people using the list to find galleries. And the first step in doing that is to recognize that it is not happening already.

    I really don't want to say much that will risk my anonymity here, but I AM a certifiable RL art/photography reviewer (as in thousands of imager a day agencies) and I still say there is a lot of junk at the top of the list that doesn't belong there and a lot of fantastic art buried. And yes, the web list is now working pretty good but the kiosk and hud are still in need of work and that is your front door.