Fleury vs. Vokoun: How Much Do Goalie Statistics Matter?

As the Pittsburgh Penguins hit the quarter-season mark with a 3-1 loss to the New Jersey Devils on Saturday, fans once again started murmuring about one of the oddest goaltending “controversies” in the NHL. With an 8-4 record, Pittsburgh is currently 2nd in the Atlantic Division and has played quite well as of late. Upon further inspection though, many people have questioned how the Penguins have arrived at their 8-4 record. Incumbent starter Marc-Andre Fleury is 5-3 with a 2.46 GAA and .906 save percentage. Meanwhile, veteran backup Tomas Vokoun is 3-1 with a 2.09 GAA and .927 save percentage. Does this have the makings of a goalie controversy?

Over the past week at The Hockey Writers, we saw both sides of what might be the most luxurious goaltending controversy in the NHL (no offense, Vancouver). In one corner, Justin Glock defended starter Marc-Andre Fleury, who seems to frequently reach scapegoat status in Pittsburgh. In the opposite corner, Billy Nauman advocated for the promotion of Tomas Vokoun, who unquestionably looks better on paper.

Who is right? Why are they right? Can anyone be right?

These are the questions I have tried to tackle over the last few days as a Pens fan. I have looked through regular goaltending statistics, “advanced” goaltending statistics, and searched for different metrics high and low to try and compare Fleury and Vokoun. Unfortunately, after many hours of research, I came to the conclusion that there is simply no adequate measure to properly compare the two goaltenders.

FLAWED STATS

When judging a goalie, most fans initially resort to three statistics: wins, goals against average (GAA), and save percentage. Each one provides a comparison point to work with, but each is also inherently flawed in its own way. Even put together, all three will never paint a perfect picture of how good a goalie actually is.

Wins

The easiest stat to assess is also the least beneficial for a goalie. Many people will say that all they care about is whether a goalie can win a game or not, but the wins category is hardly a measure of just goaltending. Some wins are earned through amazing saves while others are earned through the offensive output of the team. Obviously more wins are better, but there is no measure of how those wins are earned.

Fleury is clearly ahead of Vokoun when it comes to wins over the past 4 years, but is that a product of each goaltender or their teams? It is a mix of both that cannot be calculated properly.

Goals Against Average (GAA)

Perhaps GAA is a more useful stat because it provides a number for the average amount of goals each goalie allowed per game, but it still has its own flaws. It provides no information about a goalie’s workload, how the team in front of him is playing, or what kinds of goals are being scored. Certainly a lower GAA is better, but if a goalie is facing 20 more shots per game and has a GAA that is only .10 higher, I will take the higher GAA any day.

Vokoun wins the GAA battle for the first two years of comparison before Fleury takes over in the latter two years. However, without any information about their workload, it is hard to tell if either goalie is earning their GAA by making difficult saves to keep goals out or by letting soft goals in to inflate their numbers.

Save Percentage

When you get to save percentage, there is at least some value given to a goalie’s workload, but it still is not the complete picture. With saves and save percentage, you may know how many shots a goalie is handling and how many of those he can stop, but there is still no measure for the quality of shots or how they happened. Is one goalie facing more breakaways than another? Are there more shots off of turnovers for another goalie? These are just some of the factors that are not measured with save percentage.

Undoubtedly this stat gives a clear-cut win to Vokoun. Unfortunately, it provides no information about the defensive system each goalie is playing in and gives no number or percentage to the amount of “quality” shots they are stopping.

Many will argue that over the course of a season and with a full year of data, most factors will even out. I must firmly disagree though because different coaches implement vastly different defensive systems that have a large effect on types of chances given up. Additionally, defensive personnel will always make a large difference.

Also, should all saves be valued the same? If a goalie makes a game-changing save in the 1st period but allows a soft goal with a 5-1 lead, those stats are weighed the same as allowing a deflating goal in the first minute and stopping a pointless slapshot with a 5-1 deficit.

Current goaltending statistics are in many ways inadequate to measure the value of a goalie because of both quantifiable and unquantifiable factors that are left out.

UNUSED FACTORS

While I continued to brainstorm how to properly compare Fleury and Vokoun, I thought of other factors that are not necessarily recorded well by the NHL but would impact every goalie differently.

Blocked Shots

As the New York Rangers demonstrated in last year’s playoffs, shot blocking has become an art form in the NHL. It is an art form that can help goalies immensely and yet sometimes ruin their games completely. While a properly blocked shot is a goalie’s best friend, a missed block is often a screen or deflection that goes awry. Unfortunately, I do not know of a stat that measures screens or deflections by teammates.

Here are the blocked shots per game averages for the teams each goalie played on. Statistically speaking, there appears to be no correlation between blocked shots and the goalie stats that are normally used. Blocked shots are yet another piece of the puzzle that has not been figured in yet.

Giveaways

A defensive zone giveaway can turn into a goal in a split-second depending on where it occurs and who the puck is given to. One of a goalie’s worst nightmares can be a defensive zone giveaway where he is not set for a shot yet. There is no raw statistic that I know of that measures how many shots or goals occurs off of giveaways though.

Here are the giveaways per game averages for the teams each goalie played on. Were Vokoun’s better numbers in part because he dealt with fewer giveaways on average for the first three years of the comparison? Even this statistic does not say where the giveaways occurred or what the result of each giveaway was. Here is another part of the goaltending puzzle that we do not know.

There are plenty of other unused factors as well, ranging from defensive zone faceoffs to average amount of time the puck spent in the defensive zone. All of these stats weigh differently on goalies in different environments, but none of them have been used yet (to my knowledge).

ADVANCED APPROACHES

As I continued to search, I did find a few interesting approaches to share. None of these analyses are comprehensive but they are a step in the right direction as we continue to look for better goaltending stats.

Quality Starts

Developed by Robert Vollman of Hockey Prospectus, a quality start is determined by using save percentage and goals allowed in each game. You can read up about his methodology here or look at the statistics for every goalie over the past few years here. The point of the stat is to determine how often a goalie gives his team a chance to win the game based on the average statistics necessary for a win.

Based on the percentage of quality starts per season, it would appear that Vokoun is trending the wrong way while Fleury has played better over the past two seasons. Since this stat is based on goals and save percentage though, it is still lacking many of the factors mentioned above.

SQA

For the 2009-10 season, Ken Krzywicki analyzed shot quality and looked at factors that included shot distance, shot angle, rebounds, powerplay situations, giveaways, and shot types. His explanation and mathematical model, which admittedly is a bit over my head, can be found here. In his appendix, he includes the shot quality for (SQF) and shot quality against (SQA) that each team had and also what each team’s actual shooting percentage and save percentage were. It appears Vokoun stopped 1.2% more shots than he was expected to and Fleury stopped .5% fewer than he should have based on shot quality. It would be very interesting to see this type of data over additional years.

DIGR

Another mathematical model, “DIGR: A Defense Independent Rating of NHL Goaltenders using Spatially Smoothed Save Percentage Maps,” was created by Michael Schuckers using data from the 2009-10 season as well. His model and presentation can be found here. Schuckers used a distribution of all types of shots that every goalie in the NHL faced at different situations to create a league average distribution of shots to compare goalies with. Based on Table I of his report, Tomas Vokoun’s DIGR rating was slightly lower than his save percentage while Fleury’s was higher than his actual save percentage. Vokoun still had the higher DIGR rating for ’09-’10 though. Again, this would be an interesting model to see over the course of a few years.

These analyses demonstrate that new goaltending stats are on their way, though there is still much tinkering required to find a strong statistical comparison for goaltenders.

INTANGIBLES

Finally, there are many factors that have no statistical measure but factor into a goalie’s statistics, performance, and value.

• Defensive System
• Team Performance
• Defensive Players
• Athleticism
• Poise
• Anticipation
• Recovery Speed
• Experience
• Playoff Experience
• Etc – This list could go on for quite a while.

CONCLUSION

By now, I hope that you have realized my point in writing this editorial/article is that the arguments made by Justin and Billy can both be right, but neither of them has the complete story. I do not have the complete story either. In fact, no one truly has the complete story on evaluating a goaltender. Hockey is an incredibly complicated game and goaltending measurements just have not caught up yet. There is no method that matches up to the NFL’s Quarterback Rating quite yet.

Tomas Vokoun has stronger numbers in most statistical categories, demonstrates more poise, and has a plethora of regular season experience. Marc-Andre Fleury has decent numbers, impressive athleticism, and a plethora of playoff experience with a Stanley Cup ring.

Vokoun is likely the steadier goaltender who could push for a Vezina. Fleury has the higher highs, the lower lows, and potential to grow, mature, and improve at his age.

If I’m picking between the two, give me Fleury, his decent stats, his potential, and his Stanley Cup ring. Pure statistics won’t give you that answer, but they won’t tell you the whole story either.  (Just ask Martin Brodeur and his career .913 save percentage.)