ESPN.com – MLB/SPRING2004 – Neyer: O’s far from outstanding
Rob writes about a bunch of things today, but since my copy of Baseball Prospectus arrived yesterday I thought I’d vent a little on something he talks about that bugs me. Those are their projections.
See, I’ve been doing performance projections for ten years now, and I’ve put a lot of time into it, and I think I know what the limitations are for baseball performance projections, and as I’ve written here and at mlb.com, they’re humbling.
But Nate Silver doesn’t seem to think so. After a somewhat abbreviated look at how the BP projection system works Silver says that if it “didn’t do its job well,” BP wouldn’t use PECOTA, and then compares seven projection systems (Shandler, Diamond Mind, Baseball Primer, RotoTimes, RotoWire, Ken Warren) in hitting (OPS) and pitching (ERA), using three statistical measures.
PECOTA wins in pitching, and ties with Ken Warren in hitting, and I guess that means that Silver can claim that it “acquitted itself well,” but then he glosses over the very small differences between PECOTA (which had a .70 correlation coefficient in batting average) and the last place finisher, Rotowire (which had a .65 correlation). The other five systems ranged from .67 to .71. That simply doesn’t mean much.
Things I know about projections:
The sample you use to compare is everything. Project all the players in the game and your correlation climbs, because for most the grinding standard projection serves everyone equally well. What’s the correlation for the bulk of players? In the .65 to .75 range.
Choose correlations for a top 20 or 40 hard-to-predict players and you’ll learn more about the different projection systems, but you won’t learn which is more accurate because the sample size is too small, and one crazy good or crazy bad projection can skew results a lot, so while it will help you analyze what the system is doing, it usually won’t help you judge correlation results.
Judging projection success based on OPS is, well, one way of doing it. But as is well known, rate stats are subject to wild random swings simply based on the math and luck, and on top of that OPS doesn’t do anything to demonstrate the accuracy of playing time projetions, arguably the most important thing for fantasy players. (And, i would suggest, real baseball afficianados, too. After all, a rate is a rate, be it in 50 or 450 AB, but these mean very different things. I want to know about skills, but when it comes to the start of the season I’m looking for insight into playing time, one reason Bill James’ always interesting projections in the STATS Red Book were also always a little unsatisfying (but he did them in October, so he can be forgiven).
A few years ago Voros McCracken analysed a bunch of projections systems, also using OPS as the measure, and determined that his fledgling projection system was best. Interestingly, Baseball Prospectus’s (the old system, which was kind of crazy) was worst. Less interestingly, the correlations were all pretty close.
In Rob’s story he points to the BP and Ron Shandler projections for Barry Larkin and Ryan Freel this year. The two are essential reciprocals. One likes Freel, the other likes Larkin, by equal amounts. This is interesting because unless the players both split the difference one or the other is going to be right. But it is less interesting because it demonstrates just how muddy this whole discipline, if you can call it that, is.
As Rob points out, if both of these esteemed and wildly intelligent systems can totally disagree on two players, and as I will add, if at best they can hope to get about 70 percent right (and the bulk of that, inevitably, is because they simply found the average between the highs and the lows), what use are all these numbers (I mean, everyone is producing them now)?
I think we need them, because any competent set of projections will add to a decent portrait of what a player should be expected to do. As it gets closer to the start of the season, and a projection incorporates real playing time info, they are very helpful in picking players during a roto draft.
But trying to sell any set of projections as truly superior is problematic. Silver does something really interesting using similar players to help limn the range of possibilities for a player. As regular readers know I started doing this in 1998. But when I got excited then, and started trying to boil down that info into some usable numbers, I realized that similar players are a good way to tell a story. But there aren’t enough of them to be statistically meaningful while determining ranges of expectation.
You can tell this by looking at the similarity scores for players at www.baseballreference.com. There are some who truly are similar, and many more who are superficially so. Baseball has a long history at this point, but it has been divided into very distinct eras, across which comparisons can’t be made. Which means to project players today, you’re limited to performance from the last 20-30 years, which turns out not to be that many players.
Which, I suspect, is why those breakout, improve and decline numbers on each BP player card are so bloodless. Young players are likely to get better, while excellent players (and old ones) are not. That’s the lesson you can learn from baseball stats, and it isn’t an unimportant one, but it does undermine much of what BP is trying to do with this project.
I always seem to be bashing the BP guys, and I don’t mean to. But while a lot of what they have done and continue to do is very valuable and important, a lot of what they put out there is too much hype, not enough self-examination. My larger point is that there’s no need for you to buy projections, though you almost certainly want to look at them.