Guys,
I've been out of the office and first time to get
my email. Interesting read.
Mike does have good points, but let's not rush to
trade plans for a theoretical argument, particularly when we can't predict
likely outcome. No one really know if we are trading quantity for
quality until we've done some level of analysis on the existing data.
Outliers exist in every process. It is ill advised to react to
exceptions and use for setting general policy.
On point number one, we need to implement what
we've discussed as a priority to makes sure we discourage any large-scale
gaming. We can then evaluate if further action is
necessary.
Point number 2 seems like a reasonable request on
the surface, but will likely skew the data in favor of new issues rather that
the well established. So nothing is ever perfect. Giving the users
the ability to review votes by term (3, 6, 12 and life) would be a
nice touch.
Maher
----- Original Message -----
Sent: Monday, September 18, 2006 7:32
PM
Subject: RE: [phoenix-epic-dev] FW:
bogus rankings on EPIC
I actually think Mik
has made some very valid observations.
1) The cost of
voting on EPIC is too low. We seem to be trading off quantity vs
quality. In the long term I think this will hurt us.
Eric raised the point of requiring registrations to do a vote;
maybe this would be a cost. I also believe registration raises the
profiles of individuals (even though they might be aliases) that take part in
inappropriate behaviour. Another alternative is that a vote that
includes a well articulate comment (maybe moderated) receives extra
weight, maybe 10x votes?
2) We dont factor in
time. Opinions will change over time and we dont reflect it
here. Maybe the home page should reflect the previous 3 months but we
also have a view for the past 12 months and since
inception.
Ian
From:
phoenix-epic-dev-bounces@xxxxxxxxxxx
[mailto:phoenix-epic-dev-bounces@xxxxxxxxxxx] On Behalf Of Mik Kersten Sent: September 18, 2006 5:25
PM To: 'Mike Taylor';
'The EPIC component of Phoenix' Subject: RE: [phoenix-epic-dev] FW: bogus
rankings on EPIC
Thanks for your
comments Mike and Nathan.
Regarding this
particular case, what alerted me is that when watching the rank of Mylar, I
noticed that it and Free Marker IDE were bumped out of the top-10 by two
instantiations products within a day. Both products had mostly 9 or
10/10 ratings, but under 100. So it was easy to drop them out of
the top 10 this way. I recalled noticing the same IP adress vote
getting Data Service and Query Builder Tool out of the top 10. Then I
noticed that they had voted down non-Instantiations plug-ins multiple times,
and voted up Instantiations plug-ins multiple times. I originally
submitted this case as highly suspect. But when I saw a 3rd
vote for CodePro AnalytiX last week it crossed my threshold
for ballot stuffing and I consider the entry
bogus.
The problem is that
there is no reliable way to automatically detect such abuse, or to determine
definitely whether the above case is abuse (other than the multiple-votes
violation which the web app should detect).
I think that EPIC
ratings should play an important role in identifying helping the Eclipse
ecosystem avoid the sourceforge.net bug of having more noise than signal in
the project listings. But long term Im doubtful that the current scheme
will work. Wisdom of crowds and collaborative filtering schemes work
when voters have a non-trivial incentive to cast a well-informed vote.
Some systems use $1 payments, some reputation, and others like Amazon get it
for free by piggybacking on peoples purchases being informed. The EPIC
scheme makes uninformed votes trivial to make (e.g. someone uses the wrong
update site to install an Instantiations product and ranks it down as a
result). It also makes bogus votes trivial to make, e.g. I can sit on
one of the dozens of free proxy services like proxify.com and rate down a
product without ever being detected. Considering the relatively slow
rate of voting on EPIC, such abuse can easily skew the accuracy with which
EPIC communicates the communitys collective
perception.
I apologize for not
having mined any past discussions of this. Off-hand here are some
suggestions that could make EPIC rankings be more robust meaningful:
- Provide incentive to make an
informed ranking: require a valid eclipse.org identity for the vote (e.g.
Bugzilla/Wiki login). The downside is that will be a barrier to some,
e.g. those that dont want to take the trouble to do a quick sign-up.
But those people are much less likely to cast an informed
vote.
- Introduce a cost: require a
non-blank comment to be submitted. Single word comments can be
accepted, but for many the reputation factor introduced by (1) will be
incentive to submit informed comments.
- Factor in time: discard
rankings older than 12 months, since tools and opinions change. This
could provide a smooth migration off the old scheme onto the new scheme,
since products with a high number of good ratings wont want to discard the
old rankings.
- Consider scaling by number of
votes: the more votes, the more information there is (e.g. Netwiser should
not be above MyEclipse right now). I recall there being set formulas
for this but an expert on voting would need to chime in on choosing the
right one.
As usual with a
message this long, I feel like I should be typing it into a bugs.eclipse.org
report ;) Let me know if you create one.
Cheers,
Mik
--
Mik Kersten, http://kerstens.org/mik
Mylar Project Lead,
http://eclipse.org/mylar
From:
Mike
Taylor
[mailto:mike_taylor@xxxxxxxxxxxxxxxxxx] Sent: Monday, September 18, 2006
10:01 AM To: The EPIC component of Phoenix Cc: 'Mik Kersten' Subject: Re: [phoenix-epic-dev] FW: bogus
rankings on EPIC
I read with concern the message from
Mik Kersten. As all of you on the EPIC Council
know, I am very concerned with the legitimacy of the ranking system and am
among those pushing most strongly for making it
better.
At Instantiations we have an explicit, written policy
that prohibits our employees from ranking either our own products or those of
others on EPIC. We do this although we are fully aware that it is common
practice in the industry for those with a vested interest to vote for their
own products and against competitive products. We do this because a) we
believe our products can win on their own merit, and b) we know how angry it
makes us when a competitor unjustly slams our ratings (an unfortunately
regular occurrence).
That said, we have a couple of dozen employees and
its entirely possible that the trickle of votes Mik references came from our
IP address. We vigilantly watch for this and constantly reinforce our company
policy...but violations of that policy have certainly occurred. In every case
where we could identify the source we have appropriately disciplined the
person responsible. We will continue to do this and if Mik or anyone else can
point us to an offender we will enforce our company
policy.
Instantiations' internal policies aside, I wonder what
makes these votes inherently "bad"? Are plug-in vendors officially
restricted from voting for their own products on EPIC? Not that I know of.
What alerted Mik to these votes? Maybe if we knew we could watch that data
source and use it to enforce our own company
policies.
In Mik's data I count 17 votes over a 5 month period
involving about a dozen different products. Certainly not enough to swing
ratings much if at all. WindowBuilder, our highest volume product (and the
only one in the EPIC top 10), has literally thousands of votes, so the one in
the questionable data set had no practical effect. Our other products in the
data set have a smaller vote count, but there are still not enough votes to
affect the ratings in a meaningful way.
I have to say that even given the many discussions
we've had, and are continuing to have, at the EPIC Council about improving the
ranking system and protecting it from those who would game it...these votes
wouldn't make it onto our radar screen. What the Council is trying to do is
remove the possibility of significant, illegitimate ranking. We have to rely
on the community to alert us to small, infrequent misuses of the ranking
system, and for that I thank Mik.
I'm curious, why Mik's previous emails didn't get
the visibility they deserve? Instantiations would like to be alerted anytime
any gaming is going on with our listed products or if it is suspected that our
employees are voting in an unsavory
manner.
Regards, MikeT
PS. We would be perfectly happy if all the votes
referenced by Mik were removed from the
database.
At 9:51
AM -0400 9/18/06, Ian Skerrett
wrote:
All,
Mik Kersten sent me an e-mail
complaining about some of the rankings that have been made on EPIC. I
think we need to decide how we want to respond to Mik's
concerns.
Btw, Mik agreed for
me to post his original e-mail on the mailing list. Please copy him on
any replies, since he has not subscribed to this
list.
Thanks
Ian
Ian Skerrett
Director of
Marketing
Eclipse Foundation,
Inc.
Tel: 613-224-9461
ext. 227
Fax: 613-224-5172
ian.skerrett@xxxxxxxxxxx
http://www.eclipse.org
Blog:
http://ianskerrett.blogspot.com/
From: Mik Kersten
[mailto:beatmik@xxxxxxx] Sent: September 15, 2006 2:38 PM To: 'Ian Skerrett' Subject: bogus rankings on
EPIC
Hi Ian,
Could you give me the
email address of someone responsible for EPIC? I have submitted 3
email/web complaints of bogus rankings trying to bring up Instantiations
products and bump others near them on the top-10 down. I still have
received no response and the bogus ranking is continuing,
e.g.:
http://www.eclipseplugincentral.com/modules.php?op=modload&name=Web_Links&file=index&req=ShowRaterDetails&userid=&hostname=68.178.73.218
If these kinds of
problems with anonymous rakings are not addressed pro-actively by EPIC abuse
like this is likely to continue, and someone could blog about this and make
EPIC look untrustworthy.
Mik
--
Mik
Kersten,
http://kerstens.org/mik
Mylar Project Lead,
http://eclipse.org/mylar
_______________________________________________ phoenix-epic-dev
mailing
list phoenix-epic-dev@xxxxxxxxxxx https://dev.eclipse.org/mailman/listinfo/phoenix-epic-dev
--
---------------------------------------- Mike Taylor President and
CEO Instantiations, Inc. Power Tools for Professional Software
Developers
Voice: (503) 598-4911 mike_taylor@xxxxxxxxxxxxxxxxxx http://www.instantiations.com
_______________________________________________ phoenix-epic-dev
mailing
list phoenix-epic-dev@xxxxxxxxxxx https://dev.eclipse.org/mailman/listinfo/phoenix-epic-dev
|