Rating by Ranking

Everything online can be rated on various websites: books, movies, restaurants, videos, blog posts.. 1-to-10, 0-to-100, stars out of five, or just thumbs up or thumbs down.

The purpose is usually to create ranked lists (top 100 movies of all time, best restaurants in town, etc.), filtering out spam, or non-relevant items, or creating recommendation systems: "people who have rated items similarly to you have also rated highly the following ...", etc. etc..

In all these systems the problem that the ratings are subjective, inaccurate, inconsistent. For example someone might rate every movie with three stars, except those he doesn't like, which he rates with one star. Another user doesn't rate anything, except those he likes for which he gives five stars, etc.

A possible approach would be to ask the user to rank items instead of giving numerical ratings. For example the user looks up "The Godfather" on a movie website and notes that he has seen the movie. Then he goes to the "Goodfellas", and there instead of rating, he is asked to indicate wether it was better or worse than the Godfather. When a new item is to be rated, there would be a drop-down list with the previously seen elements ranked by preference and the user would just need to indicate where the new element goes in the list by pointing at the correct location. In practice, when the list grows too large, a fisheye-view list could appear, such as the in the demo[link]. Alternatively, the system could pick a subset of the total elements (say, 10 representative movies), and the user would just indicate where the new one fits in this smaller list. This would give less information, but it might still be enough for practical purposes.