The NHL Has a Scouting Problem

The longer I have been involved in hockey analytics the more and more I am finding myself interested in looking at the draft, the market inefficiencies and just how poorly teams have been performing at scouting to try and make the best draft selections.

Draft Situation

The draft is a really hard area for teams to succeed in.  In a given year only 210 teenagers are selected, averaging to 7 per team, while hundreds, and if not, thousands of others are passed to never be drafted or see another chance at making the next level.  Within the team’s 7 picks scouts and management need to ensure they make as many selections that will lead to NHLers as they can.  On average teams hope to select at least 1 future NHLer, but 2 or more is considered a success. This is much easier said than done because the odds of making a selection, than converts to become an NHL full-timer, is very difficult.

Being able to draft successfully in today’s NHL is very critical.  The current CBA allows you to keep your drafted players as Restricted Free Agents which in turn keeps their contracts much cheaper than equal talent in free agency.  With the Salary Cap in place teams need to stuff as many good players, and their contracts under the cap, as possible without overspending.  With the peak age of NHLers coming in at the mid-20s it becomes clearer and clearer why teams should be looking for every market inefficiency they can to help them succeed.

Scouting Experimental Data

Last summer one of my analytics colleagues Rhys Jessop started looking at how NHL teams perform at drafting.  Using the Vancouver Canucks as his model, and a very simple algorithm, he found that Vancouver’s professional scouting staff, staffed with human experts personnel traveling around the world and viewing players first hand, could not out-select players in either NHL games played or points scored.

Jessop and I partnered for a much larger experiment looking at over a decade of drafting from the position of all 30 teams.  Using metrics to determine when scouting staffs have outperformed the very simple algorithm, based on money spent and NHL games played to earn a profit, we found that 19 of 30 teams did not outperform the computer.

More recent work of ours has correlated scouts rankings and simple metrics to NHL games played.  Using Pearson’s r correlation we found that the NHL’s Central Scouting Staff rankings compared to those players’ NHL games played only had a correlation of -0.19 (or “No or negligible relationship”).  When using very simple metrics such as a prospect’s draft-year points/game, the correlation with NHL games played is much higher, at 0.41 (or “Strong positive relationship”).

This is absolutely mind blowing.  An average small-market NHL teams are spending over $2 million dollars a year on scouting.  They have staff of all sizes, full of human experts, traveling around the world for the majority of the year and watching games daily.  Yet these humans cannot do better than simple algorithms?  The return on investment is terrible

Scouting Reality

Given that scouting, on the whole,  seems to be an issue within the NHL it should be noted that it’s not always the case with every team.  There were still 11 teams who outperformed the simple model of scouting and there are those who trying to exploit market inefficiencies.  Chicago Blackhawks seem to be really invested in scouting the USHL, an area that is under-scouted.  But this is not the case for the majority of the league.

There’s a number of reasons why teams are failing to do well with their draft picks.  With the low possibility of draft picks succeeding it’s easy to see that even the best selection can fail to yield results.  However, there is also evidence that scouts constantly make the same mistakes over and over.

Size of prospects is an example of this.  It’s shown time and time ago that size of prospects is given a heavy weight in scouting opinion; larger players are over-drafted, but there’s little evidence bigger players succeed more often than smaller players.

This fetish for size happens year in and year out.  A good example of this is the Vancouver Canucks who drafted Kyle Pettit with a late round pick in the 2014 NHL Entry Draft.  Listed at 6’4″ Pettit scored 10 points in over 50 games with the Erie Otters, a team boasting high quality of teammates.  Pettit was selected for his size and his “tools” rather than his ability to excel at playing hockey.  The Canucks wasted a pick on someone, who could barely hold on to a 4th line Centre position in the OHL, while leaving the shorter Spenser Watson on the board.  Spenser Watson lead all CHL draft eligible players last year in even-strength scoring and almost went undrafted until he was selected by the LA Kings with the 209th pick.

Traditional scouting opinions is keeping the size bias alive.  Most recently ISS Hockey released their Top 30 draft-eligible prospect rankings and put Lawson Crouse 3rd, most simply for his size (currently listed at 6’4″), while playing for Team Canada at the World Juniors did not hurt.  Despite being third in the ISS rankings Crouse is scoring at a paltry rate of 0.74 points/game, 13th in the OHL of draft eligible players.  Lawson Crouse’s comparables in the CHL are: Andrew Taylor, Nick Czinder, Lucas Bloodoof, Martin Boisvenue, Joey St. Aubin, Trent Ouellete, Jean-Francois Cyr, and Cody Chikie.  If you have never heard of those players it is because amongst them they have a grand total of zero NHL games.

On the contrary, farther down the ISS list is Mitchell Marner, listed as a mere 5’11” – Marner is currently giving Connor McDavid a run for his money in the OHL.  The closest comparable to Marner is Marek Svatos; Svatos played over 300 games in the NHL scoring over 170 points.

Conclusion

Drafting is a very tough and tricky subject to deal with, but like every other area it is possibly to quantify it and analyze it to measure how teams are performing.  By moving focus away from the biggest players, and starting to focus on players who excel at hockey, teams can find market inefficiencies in their scouting to try and select future NHLers.

Scouting now is often too focused on finding the next Milan Lucic, but the Boston Bruins were lucky when they selected him.  They did not expect him to be the next Lucic.  He was one of the first players like him and there won’t likely be another for a while.  Selecting the players who excel now at hockey will give teams the higher likelihood of players who can succeed in a much tougher league.