There is a better way to evaluate relief pitchers than saves or even holds for that matter. Before we look at that metric we will need to look at a stat that is both for hitters and in this case pitchers. It is Win Probability Added (WPA), it is based on win expectancy. Quickly, win expectancy is what percentage a team has of winning a certain game at a certain time. It obviously changes throughout the game based on everything that happens, from a grand slam to a single strike. WPA captures the change in Win Expectancy from one plate appearance to the next and credits or debits the player based on how much their action increased their team’s odds of winning. Most statistics are context neutral — they do not consider the situation of a particular event or how some plays are more crucial to a win than others. We know intuitively that a home run in the third inning of a blowout is less important to that win than a home run in the bottom of the ninth inning of a close game. WPA captures this difference. You get credit based on how much your action contributes to the odds of winning, meaning a home run in a 1-1 game in the 9th is dramatically more valuable than one in a 10-1 game in the 9th. That’s why I find stats based in WPA the most informative on what happens, how important it is, and when it happens.
So now that we understand WPA we can look at a better way to measure the effect relievers have on a game. Saves tell you that a pitcher was in a game when the last out was recorded and came in when a team was up by no more than three runs. If you put in a left-handed one-out guy, up three no men on, with two outs in the ninth, he gets the same save as a guy who comes in with no outs up one with men on first and second. Did both of those pitchers add the same amount of value to the teams win? No, but they both get a save, the archaic way that “closers” get paid. It also can affect a managers decision of who to bring in a game, yes Buck Showalter that’s shade, although it did lead to one of the best bat flips in history, by not bringing in Zach Britton, because it wasn’t a save situation.
The best way I have found to evaluate relievers is on Fangraphs and it is shutdowns and meltdowns. Isn’t the most important thing is did a relief pitcher help or hinder his team’s chances of winning a game? If you answered yes, then shutdowns and meltdowns are for you. If they improved their team’s chances of winning by a certain amount, they get a Shutdown. If they instead made their team more likely to lose by a certain amount, they get a Meltdown. Relievers only earn Shutdowns if they register a WPA of +0.06 or greater in a given game. The same is true on the negative side of the ledger, earning a Meltdown if they produce a -0.06 WPA or lower during a game. Shutdowns and Meltdowns give you a way to glance at a reliever’s stat line and determine how often they register a good or bad outing. Good relievers generally have a high SD/MD ratio. Shutdowns and Meltdowns correlate very well with saves and blown saves; in other words, dominant relievers are going to rack up both Saves and Shutdowns, while bad relievers will accrue Meltdowns and Blown Saves.
You should keep in mind that not all Shutdowns (or Meltdowns) are created equally. A +0.40 WPA game and a +0.07 WPA game each earn a single Shutdown, to say nothing of the fact that Win Expectancy is based on average teams in average situations, not the two specific teams in the specific contest. The best three hitters on a team with a platoon advantage are a much taller task than the bottom of the order, but that information isn’t included when looking at Shutdowns and Meltdowns. Shutdowns and Meltdowns are a simple way to determine whether or not the pitcher had an effective outing or not. They don’t tell you if a pitcher pitched well, but they do tell you if the team had good outcomes when the pitcher was on the mound in a particular game. The results won’t tell the WHOLE story, but it gives you a better picture than a save or blown save. At the end of the season, it is a good way to see, like any stat, how a pitcher performs over a full season, and I typically avoid it in small sample sizes.
The +/- 6% cutoff puts SDs and MDs on a similar scale as saves and holds, meaning 40 shutdowns is roughly as impressive as 40 saves or 40 holds. Dominant closers or set-up men will typically have 35 to 40+ shutdowns and a handful of meltdowns. Meanwhile, meltdowns are more common than blown saves, and they can happen to both closers and non-closers alike. The worst relievers will rack up around 10 to 15 meltdowns in a season.
Thanks as always for reading and we will be running these pieces all week for what we are calling advanced stat week. Let us know what you would like to see me tackle or if you have any questions. You can ask or just follow us on twitter @SidedSox, and follow us on Facebook. You can also email us at SoxSided@Gmail.com.