This year’s guide is at the printer, and should start showing up in stores soon. Here’s the front cover.
Projections
The Forecasters Challenge 2011: We have a winner!
Yes, in this quirky little game that Tom Tango has put together over at the ever enjoyable and challenging insidethebook.com blog, Ask Rotoman won the official Best Projections of 2011 competition, edging out the Consensus picks of all 22 forecasters (as well as beating the 21 other forecasters, as well).
You can read Tom’s post about the competition, which is for the most part his way of trying to demonstrate that the value added of a “projection system” over the weighted averages he uses for his Marcel the Monkey projection are slight. There is another side to that story, but we’ll leave that quarrel for another time.
The bottom line is that projections take many forms, for a variety of distinct purposes, and no one has come close to cracking the rather substantial variance in player performance that can only be attributed to luck (or unluck). I make projections for my own use, because I need to know what’s going into them, and I offer them to customers because they ask for them. I hope that’s because they trust that what I’m putting into them is the best stuff we have to work with. This year it turned out that the Challenge agreed, which is nice.
Congratulations to Consensus, RotoWorld and KFFL, each of which won one of the unofficial contests, and to Consensus and RotoWorld, which finished high atop the z-score derived standing for the four combined contests.
Forecasters Challenge
Tom Tango runs a neat little competition at The Book blog. He’s just published year’s first results for the three unofficial contests and the official one. My projections were in the top quarter in 2009 and are there again this year.
It’s exciting, even if the takeway is that our Consensus picks are better than any individual forecaster’s.
The Accuracy of Projections–the hitting optimizer
I participated in my first auction last night, the Cardrunners League, and because we’re using CBSsports.com to run the league, you can easily get a chart with the projected stats for each team. I did that and learned that according to the CBS projections my hitting is mediocre (uh-oh, and they’re not as negative on Grady Sizemore as they probably should be) and my pitching is pretty good. Overall, it looks like 75 points or so for my team, which I’ll take.
But then I found on the CBSsports.com site, a story by Al Melchior and a widget that lets you graphically compare the CBSsports projections and the Accuscore projections. The differences are striking, and a good reminder that projections give you a very limited amount of information.
You can find out more about my draft at Patton and Co, in the Kevin Gregg discussion.
Marcel vs. the others
Having just finished and released my projections for the Patton $ Online Software product I’m thinking about the accuracy and usefulness of projections more than usual (and I usually think about this subject a lot).
Those of us who make projections want our projections to be the most accurate, but it turns out that measuring a set of projections versus what actually happened is a complicated business. Just how complicated becomes clear if you read the first two parts of Tom Tango’s analysis of five different projection systems from 2007-2010.
But you don’t have to, Tom says you can skip those parts, and you’ll still appreciate the results, which show that CHONE was probably the best projection system in recent years, but that it wasn’t much better than Marcel, which Tango invented as a simple baseline projection that could be measured against more sophisticated systems to evaluate them. If they don’t do better, they aren’t adding value.
The question is how much value any of the systems is adding. The answer depends on what you’re looking for, but the assertion by one of the commenters that accurate projections probably matter most to fantasy players rubs me that raw way. As the survey results show, using projections to value players for your fantasy league isn’t going to get you very far. The margin of error for each projection is far wider than the range of projections from all the various sources.
Different projection systems incorporate different aspects of baseball analysis. My projections use complex regression analysis of previous performance, filtered first by age, and then by my tweaking.
Other systems use other inputs. PECOTA draws on similar player/career arcs to project into the future, for instance, while ZIPS and CHONE incorporate some of the newer stats to establish complex systems of regressing outlying performance to the mean.
I have my doubts how far such empirical formulation will take us toward the grail of accurate projections, the ball hasn’t moved much in recent years despite lots of new data, but all the work is necessary to tease out what real information there is to be found in the numbers. Tango’s report and the many comments that follow it are invaluable for showing what the challenges are, and perhaps eventually suggesting a way forward.
Fan Projections Engine Live at Fangraphs
You can enter your own player projections at Fangraphs now, and join the crowd sourced set. The link takes you to Ben Zobrist, but you can project any player once you register.
Jason Collette has made a list of links to the Tampa Bay Rays players you’ll want to be projecting this year.
Tom Tango’s Forecaster’s Challenge: Final Results
You can read Tom’s recounting here.
My projections scored in the top half in 2009, but fell off this year, ending up in the lower part of the middle group. I haven’t broken down what happened yet, but reading through the comments and results it seems that using the community playing time forecasts in the future might be a good idea. Many of the leaders are using either the Fangraphs or The Book playing time forecasts, capitalizing on the crowd’s ability to incorporate localized information. This isn’t a luxury we have in February, but as this information becomes available in March, it makes sense to make more use of it.
I’d like to point out that for the second year in a row John Eric Hanson won. Congratulations John!
Illustrating the Projection Problem: Real Life Example
Last post I wrote about how accurate projections have to regress player performance to the mean. I used the example of AB, since it is simply a measure of playing time and role, not subject to the variance that Hits is, for example.
Here is a chart showing the Top 10 2009 AB Leaders, and how they did in 2010.
Projections are not prices, Part 1
PROJECTIONS ARE NOT PRICES, Part 1
Winning at playing fantasy baseball has two obvious components:
Player Projections and Player Pricing.
It is, one assumes, most helpful to have the best projections, because they tell us what players are going to do. The best set of projections would give you the best idea of who is going to be good this year, and who is going to be not so good, and this information should give you an edge over someone who doesn’t have such good projections (or no projections at all).
Plus, good projections should lead to better prices. If you know better than anyone else what the players are going to do in the coming year, you should be better able to value a home run, for instance, in the context of all the other home runs hit, and so on and so forth for all the categories. This would give you a better price in each category for each projection and overall more accurate prices for all players.
This is how good projections are thought to lead to winning fantasy teams, but it just isn’t so. At least not when it comes to the conversion of projections into prices to pay at auction. The fact is that accurate projections are a map of regression to the mean. In making accurate projections we average out the highs and lows of a player’s history, in order to better identify his baseline, which is the core description of his true talent.
A perfect illustration of this comes from the projection of at bats. In any given year six to 10 hitters will accumulate more than 700 PA. These are, obviously, guys who have and hold the leadoff position in the lineup, on good teams, all year long. But when one uses regression analysis to look at past history of players with more than 700 PA in a year, the math comes back that that sort of player will have 630 PA in the subsequent year.
What the formula does is look at the, let’s say, 10 hitters with 700 PA each (for a total of 7000 PA), and notes that on average in subsequent years a player in that group will have, on average, 630 PA. Now this could break out in a variety of ways. Nine might have 700 again, and one 0, or 5 might have 700 and five might have 560. The specifics are changeable, but the point is that based on the actual history of baseball players over the past 40 years or so, what we know is that on average each of the top 10 PA guys in one year will have 10 percent fewer at bats the next year.
What we also know, is that most of the leading PA guys in one year will be the leading PA guys the next year, with about 700 or more.
And what we don’t know is which player or group of players is going to fail and bring down the average PA of the group.
So, is a good projection the one that gives each of the 10 players 630 PA, spreading the risk between them?
Or is a good projection one that gives each of the 10 players 700 PA, getting more of them individually right, but making the misses that much more wrong?
More Cardrunners Debate, at THTFantasy.com
In a previous post I wrote about the Cardrunners League I’m playing on, pitting quants vs. so-called fantasy experts. This has become a rather unwieldy mess, in part because the central issues keep erupting into flashes of debate about whether analysis or intuition matters more. The funny thing is that even when there is too much blather in this pissing match, there are interesting issues that come up about what we know and what we don’t know about the game of fantasy baseball itself.
Now, some THTFantasy writers are weighing in at their own site. Derek Carty is also a Cardrunners League competitor, but I like Derek Ambrosino’s take, which makes many of the points I’ve been trying to make, often with more wit. Derek also quotes a Mike Podhorzer piece about what makes an expert, which is a must read. Paul Singman also talks about the problem of identifying which players and which fantasy strategies actually work, which is certainly a huge issue. How do you decide what works if there’s no definitive way to test it?
For my part, I would love a tool that let me test different strategies in thousands of runs, to see what range of possibilities there really are. But I think the Derek defines the nature of the game in a most instructive way when he compares it to chess (a head to head game) and the stock market (a one against many game with many winners and many losers). Roto is a game of one against many with only one winner, which is different. Setting yourself apart would seem to be essential to win, but how is this done? The quants seems to think incrementally, by buying value. I think the so-called experts see more need for radical action, though it is certainly open to debate whether these are genius picks or zagging while others zig. All in all, a fascinating discussion if you have the time.