Skip to main content
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
RE: [phoenix-epic-dev] FW: bogus rankings on EPIC
Title: Re: [phoenix-epic-dev] FW: bogus rankings on EPIC

Thanks for your comments Mike and Nathan. 

 

Regarding this particular case, what alerted me is that when watching the rank of Mylar, I noticed that it and “Free Marker IDE” were bumped out of the top-10 by two instantiations products within a day.  Both products had mostly 9 or 10/10 ratings, but under 100.   So it was easy to drop them out of the top 10 this way.  I recalled noticing the same IP adress’ vote getting “Data Service and Query Builder Tool” out of the top 10.  Then I noticed that they had voted down non-Instantiations plug-ins multiple times, and voted up Instantiations plug-ins multiple times.  I originally submitted this case as “highly suspect”.  But when I saw a 3rd vote for CodePro AnalytiX last week it crossed my threshold for ballot stuffing and I consider the entry bogus.

 

The problem is that there is no reliable way to automatically detect such abuse, or to determine definitely whether the above case is abuse (other than the multiple-votes violation which the web app should detect). 

 

I think that EPIC ratings should play an important role in identifying helping the Eclipse ecosystem avoid the sourceforge.net bug of having more noise than signal in the project listings.  But long term I’m doubtful that the current scheme will work.  “Wisdom of crowds” and collaborative filtering schemes work when ‘voters’ have a non-trivial incentive to cast a well-informed vote.  Some systems use $1 payments, some reputation, and others like Amazon get it for free by piggybacking on people’s purchases being informed.  The EPIC scheme makes uninformed votes trivial to make (e.g. someone uses the wrong update site to install an Instantiations product and ranks it down as a result).  It also makes bogus votes trivial to make, e.g. I can sit on one of the dozens of free proxy services like proxify.com and rate down a product without ever being detected.  Considering the relatively slow rate of voting on EPIC, such abuse can easily skew the accuracy with which EPIC communicates the community’s collective perception.

 

I apologize for not having mined any past discussions of this.  Off-hand here are some suggestions that could make EPIC rankings be more robust meaningful:

  1. Provide incentive to make an informed ranking: require a valid eclipse.org identity for the vote (e.g. Bugzilla/Wiki login).  The downside is that will be a barrier to some, e.g. those that don’t want to take the trouble to do a quick sign-up.  But those people are much less likely to cast an informed vote.
  2. Introduce a cost: require a non-blank comment to be submitted.  Single word comments can be accepted, but for many the reputation factor introduced by (1) will be incentive to submit informed comments.
  3. Factor in time: discard rankings older than 12 months, since tools and opinions change.  This could provide a smooth migration off the old scheme onto the new scheme, since products with a high number of good ratings won’t want to discard the old rankings.
  4. Consider scaling by number of votes: the more votes, the more information there is (e.g. Netwiser should not be above MyEclipse right now).  I recall there being set formulas for this but an expert on voting would need to chime in on choosing the right one.

 

As usual with a message this long, I feel like I should be typing it into a bugs.eclipse.org report ;)  Let me know if you create one.

 

Cheers,

 

Mik

 

--

Mik Kersten, http://kerstens.org/mik

Mylar Project Lead, http://eclipse.org/mylar

 

 

From: Mike Taylor [mailto:mike_taylor@xxxxxxxxxxxxxxxxxx]
Sent: Monday, September 18, 2006 10:01 AM
To: The EPIC component of Phoenix
Cc: 'Mik Kersten'
Subject: Re: [phoenix-epic-dev] FW: bogus rankings on EPIC

 

All,


I read with concern the message from Mik Kersten. As all of you on the EPIC Council know, I am very concerned with the legitimacy of the ranking system and am among those pushing most strongly for making it better.

At Instantiations we have an explicit, written policy that prohibits our employees from ranking either our own products or those of others on EPIC. We do this although we are fully aware that it is common practice in the industry for those with a vested interest to vote for their own products and against competitive products. We do this because a) we believe our products can win on their own merit, and b) we know how angry it makes us when a competitor unjustly slams our ratings (an unfortunately regular occurrence).


That said, we have a couple of dozen employees and its entirely possible that the trickle of votes Mik references came from our IP address. We vigilantly watch for this and constantly reinforce our company policy...but violations of that policy have certainly occurred. In every case where we could identify the source we have appropriately disciplined the person responsible. We will continue to do this and if Mik or anyone else can point us to an offender we will enforce our company policy.

Instantiations' internal policies aside, I wonder what makes these votes inherently "bad"?  Are plug-in vendors officially restricted from voting for their own products on EPIC? Not that I know of. What alerted Mik to these votes? Maybe if we knew we could watch that data source and use it to enforce our own company policies.

 

In Mik's data I count 17 votes over a 5 month period involving about a dozen different products. Certainly not enough to swing ratings much if at all. WindowBuilder, our highest volume product (and the only one in the EPIC top 10), has literally thousands of votes, so the one in the questionable data set had no practical effect. Our other products in the data set have a smaller vote count, but there are still not enough votes to affect the ratings in a meaningful way.

 

I have to say that even given the many discussions we've had, and are continuing to have, at the EPIC Council about improving the ranking system and protecting it from those who would game it...these votes wouldn't make it onto our radar screen. What the Council is trying to do is remove the possibility of significant, illegitimate ranking. We have to rely on the community to alert us to small, infrequent misuses of the ranking system, and for that I thank Mik.


I'm curious, why Mik's previous emails didn't get the visibility they deserve? Instantiations would like to be alerted anytime any gaming is going on with our listed products or if it is suspected that our employees are voting in an unsavory manner.

Regards,
MikeT

PS. We would be perfectly happy if all the votes referenced by Mik were removed from the database.

 

At 9:51 AM -0400 9/18/06, Ian Skerrett wrote:

All,

 

Mik Kersten sent me an e-mail complaining about some of the rankings that have been made on EPIC.  I think we need to decide how we want to respond to Mik's concerns.

 

Btw, Mik agreed for me to post his original e-mail on the mailing list.  Please copy him on any replies, since he has not subscribed to this list.

 

Thanks

Ian

 

 

 

Ian Skerrett

Director of Marketing

Eclipse Foundation, Inc.

Tel: 613-224-9461  ext. 227

Fax: 613-224-5172

ian.skerrett@xxxxxxxxxxx

http://www.eclipse.org

Blog: http://ianskerrett.blogspot.com/ 

 

 

 

 

From: Mik Kersten [mailto:beatmik@xxxxxxx]
Sent: September 15, 2006 2:38 PM
To: 'Ian Skerrett'
Subject: bogus rankings on EPIC

 

Hi Ian,

 

Could you give me the email address of someone responsible for EPIC?  I have submitted 3 email/web complaints of bogus rankings trying to bring up Instantiations products and bump others near them on the top-10 down.  I still have received no response and the bogus ranking is continuing, e.g.:

 

http://www.eclipseplugincentral.com/modules.php?op=modload&name=Web_Links&file=index&req=ShowRaterDetails&userid=&hostname=68.178.73.218

 

If these kinds of problems with anonymous rakings are not addressed pro-actively by EPIC abuse like this is likely to continue, and someone could blog about this and make EPIC look untrustworthy. 

 

Mik

 

--

Mik Kersten, http://kerstens.org/mik

Mylar Project Lead, http://eclipse.org/mylar

 


_______________________________________________
phoenix-epic-dev mailing list
phoenix-epic-dev@xxxxxxxxxxx
https://dev.eclipse.org/mailman/listinfo/phoenix-epic-dev

 

 

--

----------------------------------------
Mike Taylor
President and CEO
Instantiations, Inc.
Power Tools for Professional Software Developers

Voice: (503) 598-4911
mike_taylor@xxxxxxxxxxxxxxxxxx
http://www.instantiations.com

Back to the top