Posted by: Paul Hewitt | February 8, 2012

Prediction Market Design Issues

I’ve been taking part in the Good Judgment Project, which is attempting to predict global events.  I’ve discussed some of the quirks in the design of the prediction markets.  Here’s a few more.

Initial Likelihoods

Most of the questions involve binary events, but every once in a while there are multiple options in winner-take-all markets.  A recent example is this question (click to expand):

Note that there are five possible outcomes.  When the question was first posted (February 7, 2012), each of the outcomes was given a 20% likelihood of coming true.  As a result, early purchasers (like me) could purchase an outcome that would pay off relatively handsomely.  I was able to pick a very likely winner, simply by acting swiftly.  One very quick Google search yielded enough information to select (b) Henrique Capriles Radonski as the overwhelming favourite.

Of course, the same thing happens with binary markets, where the initial likelihoods are set at 50%, meaning you can double your money by being the first to pick the right outcome. The problem is exacerbated when there are multiple contracts in the market.  In this case, I will quintuple my investment, if correct.

The issue is that some participants can achieve significant “profits” without actually having much real knowledge or superior predictive abilities.  One of the purposes of a prediction market is to identify the best predictors – they end up with the most money, allowing them to have a greater say in future predictions.  In cases such as these, there are windfall gains to be had by the earliest traders, because the initial odds are not set properly.

The odds are set, assuming no information is available to make a decision.  That is, it’s kind of like a five-sided die roll or a coin flip for binary events.  In the above case, there is information available, which should have been taken into account in setting the initial odds.  We can assume there was quite a bit of “good” information about the outcome, because within one day, the likelihood of the frontrunner jumped from 20% to 92.8%!

Trade Limits

Another quirky design issue involves limiting trades to $1,000 per market.  This restricts someone with superior knowledge from having the appropriate amount of influence in a market.  To my way of thinking, this runs counter to prediction market theory.  Essentially, these market quirks preclude a successful trader from being able to justifiably influence any market, but it does allow him or her to invest in more markets than unsuccessful traders.  Of course, the early winners will be able to maximize the number of markets in which they can make the maximum bets.

Short-term vs. Long-term

Another interesting observation is that new questions are added every few weeks.  While I have done reasonably well, I still find that I am almost fully invested, without being able to take positions in all of the markets I would like to.  When new questions arise, I have to decide whether I wish to get out of some markets in order to invest in new ones.  This involves deciding whether it is better to get out of a long-term question (say 3 – 12 months hence), which may pay off handsomely, to be able to invest in one or more shorter term markets, which may collectively pay off even better.

In other words, you have to keep your money “working”.  Long term investments are riskier (more unpredictable intervening events may occur) and tie up your money.  So, even though you may have superior information about a market outcome, it may not be financially appropriate for you to act on it and place a bet.  This provides incentives for traders to invest where they have the best information relative to other traders, even though they may have better information than other traders in many (or all) markets.

This is probably a good thing, but, by design, the overall exchange is leaving “good” information on the table!



  1. on Knew The News, to prevent making too much profit from unbalanced markets, trading is limited to buying only (so you cannot cash in rapidly) and the creator of the market is banned from making predictions himself for two hours.

    this enables the community to check the initial odds and probably suspend and eventually void the market if they are substantially wrong, too far away from general expectations.

    though, what establishes “too far”? and what actually are to be considered “general expectations”?

    Knew The News does not limit bet sizes below 100,000. but when predictions are made exceeding 100$, the return value will already be influenced through the market maker (Hanson, LMSR). this encourages new players since smaller bets pay out relatively better.

    How does your market maker reflect the situation in which one of the outcomes renders impossible?

  2. I’m afraid that I’m not privy to the Good Judgment Project’s market design. I’ve asked questions, including how superior predictions are determined, but haven’t heard back yet.

  3. I understand this experiment is government sponsored. there is a large range of prediction markets available right now, both public and commercial. but I can imagine that they build their own system.

    from your expertise, how would you handle these situations? any prediction market needs to somehow cover these issues and impose mechanisms to keep the game fair. and since you are someone who has a lot of experience and knowledge about prediction markets in general, I assume you have your own opinion on these?

  4. First, I think the initial odds should be set by a panel of “experts”, based on all available information. We “know” that the crowd is smarter than the “experts”. So, let’s make them prove it, by improving the accuracy of the prediction. This would take away the windfall effect.

    The trade limit is not as big a problem, because most traders won’t be able to invest in as many markets as they may want.

    In order to increase liquidity in longer term markets, I would suggest that each trader have a short-term fund and a long-term fund. Not only would it encourage more long-term trading, but it would allow us to see whether there is a difference in relative accuracy.

    I’m not a big fan of the market allowing you to cancel your trade anytime before the close (and get all of your money back). I think this encourages risk seeking behavior. Also, why can’t you earn a profit by buying low and selling high (before the market closes)?

    When trades are placed, the market quotes do not change right away. I’m not sure how they are doing it, but it does not appear to be a strict LMSR.

    Those are the main ones, off the top of my head. Please feel free to add any additional comments. It would be nice if the project coordinators commented!

  5. from my point of view, there it isn’t probable to set the “correct” starting odds. this would assume that actually all available information is in reach of these experts, and these experts were able to weigh these information accurately. that is similar to the rational choice theorem, which is a need theory, but practically far from achievable.

    personally, I advice any market creator to create markets on personal expectations in a way that he himself could not decide in which way to bet would be more promising. that way, the market odds would be correctly balanced.

    a second mechanism is to make trades have larger influence on the odds when they are made in early stages after creation. that way, the correction of the odds happens faster, and the advantage gain for early predictors based on slightly incorrect odds would be less.

    I agree that selling should be possible before settlement. IMO that would be the only way that markets could actually shift odds according to real life changes in the situation covered by the market.

    but how would you distinguish short and long term predictions? where does long term start? surely, to encourage betting on long term markets, some kind of incentive is needed. but I’m not sure how this should work in a mixed environment.

  6. There are lots of incentive problems with longer-term markets – longer time until the outcome is revealed (boredom), longer holding period (opportunity cost of funds invested), especially if competing with shorter-term investments. I don’t have the answer to this, other than to segregate ST and LT investment funds.

    For some markets it is relatively easy to set the initial odds – e.g. political markets could use latest poll results. I don’t think it has to be particularly accurate, just a lot better than 50:50 for binary contracts (unless it truly is a toss up). I think any reasonable, hopefully unbiased, estimate would work for the initial odds.

  7. one idea could be to use some kind of multiplication based on the age of the prediction at settlement. lets say: the net calculated winning times the age of your prediction in days is what you get in return. this would be a great incentive to make long term bets. but wouldn’t this make predicting too overly complicated? and in addition, you couldn’t make a suggestion about expected returns in case of winning …maybe you could declare this to be an additional bonus.

  8. That’s an interesting blog post on prediction vs. trading. What I learned from my experience in GJP is actually that prediction (GJP style) is actually quite different from trading.

    As you say, trading is mainly about getting ahead of the market, timing and bet allocation. In our group decision, GJP doesn’t reward you for being ahead of your time because in our group, we get a time weighted score. Your great prediction 1 day ahead only gets you 1 day of scoring benefit. So in my group, it’s more rewarding of consistency of beliefs over a period time.

    However, I’m not convinced changing the system to reward timing would improve accuracy of the prediction or the rating of predictors. Like you say, it allows people with no predictive ability to get in early to exploit the incorrect allocation at the beginning.

    So perhaps that is one of the end goals of the GJP study is to figure out what sort of scoring/trading systems are best at 1. creating accurate predictions, 2.rating predictors or 3. increasing participation. They may be in tension.

    My theory on actual investment markets is that the end goal is 3. increasing participation (or more accurately, $ trading volume). Being good at 1 or 2 is only a side effect.

  9. Yes, I agree, increased participation is one key to achieving accuracy, but there must also be an increase in total market information content as a result. Incentives are used to encourage information search and new models to generate better (more accurate) predictions.

    Can you explain your reward method? I’m a bit confused as to how it rewards “accuracy”. I should note that I’m confused as to how my group determines accuracy, too! I mean, if a low probability event does occur, does it mean the prediction was inaccurate (especially for binary markets)?

    How does your group determine consistency of beliefs? Is a belief consistent if it is maintained, despite new information?

    Anyway, it is an interesting project!

    Thank you for your comments.

  10. I think your group design rewards being “ahead of the curve” more than my group design. As I posted in the comment to your last post, our group does a day by day averaging of errors. So if you know the news 1 day ahead of time, you win 1 day’s error differential. The next day, what you know will be public, and everyone else will adjust their prediction accordingly.

    While in your group design, if you’re ahead (for example, predicting that Capriles will have a scandal and drop out), you can profit 10x by knowing this 1 day ahead.

    There are obviously pros and cons of setting it up each way, and I suppose the study will tease them out.

    For what it’s worth, I think my day by day group design gives a more accurate continuous reading of the probability. It rewards timely and diligent updating after news items. However, it does not reward taking a chance on an outlier opinion a day or 2 ahead of time, so it’s not that effective as an “early alert system”. We’re really good at being a lagging indicator.

    Your group design rewards being ahead of the curve. It’s riskless to take a gamble, so if you think Caprile about to run into a scandal, you can make a low risk bet against him. However, it’s not as accurate in terms of a continuous reading because there’s no incentive to make further trades once you’ve locked in a favorable 5 to 1 early bet.

    Despite that, I like the design choice not to allow individuals to cash in the market price on Capriles before the end. Unlike a stock market, there’s an ultimate outcome here. So if you allow people to sell out early, you’re incentivizing to predict people’s sentiments (just like in a market) rather the ultimate outcome.

    To me its an interesting attempt to remedy one of potential issues with standard prediction markets. Maybe they have yet another group that does a standard market mechanism, and they’ll compare your group’s results vs. that group’s results.

  11. […] second season.  Again, we will be predicting events around the world.  Last year, my group used a modified prediction market.  It was a bit unusual, because you could rescind earlier predictions and lose nothing.  In my […]

Leave a Reply


You are commenting using your account. Log Out / Change )

Twitter picture


Facebook photo


Google+ photo


Connecting to %s


%d bloggers like this: