Binary decisions (updated)

I like this: You provide two answers for some problem you can’t solve and other users of the website are asked to choose for you. Meanwhile you make five choices for problems provided by others. Once you’ve finished, you get a nice barchart.  You can head back to check up on progress if you remember not to lose the URL.

Fantastic idea, except we don’t know much about the people who use the service. For instance I’m not sure how good they are arithmetic…

Now I wonder how to use this to work out how good the others are at making decisions before trusting them to make a decision.

Update: it’s looking better now.  26 votes, 69% were correct, almost above chance (by the binomial test, p = 0.08).  Maybe we need around 50 people to use this thing to do arithmetic for us.