Back in January, I wrote a piece about the PECOTA projections for 2015. I said to take those with a grain of salt, and rightfully so. The only team that could match their projection is the Baltimore Orioles, as they're on pace to win 79 wins; their preseason projection said 78 wins.
As I've mentioned many times, these projections are guidelines. They usually do very well on the aggregate, but individual teams often have a great deal of variance. They also miss qualitative analysis that isn't possible to put in an algorithm. Nonetheless, they're still fascinating tools, and they do an admirable job of trying to strip bias from baseball analysis. But in this case, they are likely wrong by nine games. Where did they go wrong?
In terms of position players, the first, and most obvious reason, is Alex Rodriguez. PECOTA believed that he would be worth a mere 0.8 WARP, but instead he's been worth 2.5. It's understandable, though, that a projection system would make this error. Heck, no one could have guessed he would be this good at the plate, so it's no surprise that a very conservative system would not make this prediction. His 90th percentile projection still doesn't match his 2015, so the 50th percentile wasn't going to be close.
Another obvious reason is Mark Teixeira. Even though his most recent injury derailed his great comeback, he still beat his projection by 1.5 WARP. This, like with Rodriguez, was surprising. Teixeira has had an extensive injury history over the past couple of years, and even his healthy play has deteriorated. His very successful season, according to PECOTA, was an 80th percentile outcome.
And finally, Didi Gregorius was another, albeit more surprising, miss. This one is a little more forgivable; Gregorius was a player with a history of relatively poor offensive performance, and it's not like he had an illustrious career like with Teixeira, for example. PECOTA has missed his projection by about a win, and they list that as an 80th percentile outcome.
In terms of pitchers, we'll start with the obvious. One of the most egregious mistakes, one that I even pointed out back in January, was that of Dellin Betances. PECOTA projected he would be worth 0.3 WARP, and we all knew that was just ridiculous. Relievers are fickle creatures, but no one in their right mind would say he'd be replacement level. But in this system, they factored in his poor starting performance and improperly weighed it against him. Editor-in-chief Sam Miller recognized this shortcoming, and we see it here. He has instead been worth 2.6 WARP.
Nathan Eovaldi was also a huge miss, and by about 2.5 WARP. Even though he will miss the rest of the season, he has bested even his 90th percentile projection. This is one of those cases where the qualitative bests the quantitative; we all knew at the time of the trade that his stuff was excellent and he had yet to put it together, but PECOTA didn't buy any of it. They also didn't consider an excellent pitching coach, too.
There's a miss in the negative sense, though, and that was CC Sabathia. PECOTA earnestly believed he would be worth 1.5 WARP, and it was a reasonable belief! Sabathia has a near-Hall of Fame career, and a lot of bad luck compounded with injuries in recent years. He has instead been near to replacement level, and I can't imagine the 2016 projections being as generous.
As one can plainly see, a lot of the discrepancies between the projections and the actual results were very fluky. Both Mark Teixeira and Alex Rodriguez looked to be near the end of the line, but instead they have put up a combined 5.6 WARP, instead of the projected 2.4. I don't know of any person, or algorithm, that could have predicted that. In Rodriguez's case, it's without historical precedent. Even the Eovaldi and Gregorius projections could be argued to be reasonable, albeit maybe not that negative. Only in the case of Betances was it outrageous at the time.
Many of the other projections are incredibly close. They're not perfect, but the projections for just about everyone else are in the ballpark, so you can't say it's all wrong. The point of these is to get a general idea of talent level, and it did that. 80 win projections can easily lead to 60 win results, but in this case things broke in the right direction. The Yankees breathe a sigh of relief.