Christopher Ralph is a hockey writer with a focus on prospects and the entry draft, as well as the Leafs’ correspondent here at THW.
In grading which scouting service and/or ranking system made the best predictions or was most accurate in terms of how the NHL 2009 entry draft went down, there are many limitations. It will likely be somewhere between three to five years before we can begin to understand which scouting/ranking system was most accurate in their prognostications, and compare this to how NHL team’s professional scouting teams performed.
However, until time elapses and players develop, the only true comparator we have is to compare in some fashion the various rankings to how NHL teams actually drafted. Again, there are limitations to such analyses and I will try and outline some of these when discussing some of the comparisons I performed below.
First off, I did a straight up count of which of the main rankings out there got the most correct in the 1st round out of the sixty-one prospects drafted.
*The limitation of this analysis is that the various rankings systems out there are not necessarily predicting when a player will be picked, but ranking players based on their own scouting of the respective prospect.
Here are the results:
1. Bob McKenzie (TSN.ca): 54/61
As TSN states: “This is not a prediction of which team will take which prospect. In fact, when TSN puts its rankings together it doesn’t even take into account the order of selection at the draft. It is a ranking based on consensus opinion from 10 NHL scouts surveyed by McKenzie, who has been using the same process and formula for draft rankings since he originated The Hockey News Draft Preview in 1984, over 25 years ago.
And neither is it the personal or subjective opinion of McKenzie, who instead relies on the combined expertise of the scouts surveyed to calculate where prospects should be slotted on consensus from Nos. 1 through 60. Historically, it has been a good barometer of which players will be chosen in the first round.”
2. The Hockey Spy’s Draft Preview/2 Round Mock: 51/61
*My rankings and published here at THW.
3. (tie) McKeens: 49/61
3. (tie) THN: 49/61
5. Red Line Report: 48/61
6. Future Considerations: 47/61
7. International Scouting Services: 45/61
*Central Scouting Bureau/Services were not listed due to their segregated rankings. They separate North American skaters from European and also have skaters and goalies ranked separately.
Secondly, I went through each player taken in the first 2 rounds that was applicable to most rankings, and ranked each of the major scouting/ranking services (from 1 to 7) based on how close the predicted ranking was to where the player actually went in the 2009 NHL entry draft.
Limitations:
*This was by no means a sophisticated statistical analysis, but serves as a decent comparison between the rankings.
- Bias: of course, I did the analysis, so there may be my bias, although the numbers do not lie.
- The future: possibly in 3 to 5 years, based on how prospects perform, the various rankings could prove more accurate. But: the future is the future. It will be cool to look back after “X” number of years and see how the rankings match up then. Of course, scouting services perform their rankings as a projection of how the prospects will develop and the players they will become over the next 3 to five years.
- A couple players were left off if all of the rankings totally missed the boat, as it was next to impossible to analyze.
- Central Scouting Bureau/Services again were not listed due to their segregated rankings. They separate North American skaters from European and also have skaters and goalies ranked separately.
- ISS also separates out goalies, so in most cases I had to extrapolate, but I did give them the benefit of the doubt in these occasions.
The final totals calculated are listed in brackets below. In this analysis, the lower number indicates higher accuracy in terms of rankings closeness to where a respective prospect actually went in the draft.
1. The Hockey Spy (144) – Just edging out TSN and their 10 NHL scouts.
2. Bob McKenzie – TSN.ca (146) – An extremely close second – admittedly, with numbers this close, it likely could go just as easily in TSN’s favour in a more detailed analysis.
3. Future Considerations (156) – A very respectable and great score by FC!
4. Red Line Report (191) – Kyle Woodlief and crew finish a distant fourth.
5. THN (204) – The Hockey News are holding out hope their rankings look better five years from now.
6. McKeens (210) – well back in the pack!
7. ISS (224) – Someone had to bring up the rear!
I plan to do a more detailed analysis which I will post in the near future and may provide an even clearer picture.
Again, the significance of these analyses is unknown, but at present time it at the very least gives us some form of comparison. Given the limitations noted above and the fact I performed the analyses, I welcome and would appreciate anyone who wishes to perform a separate analysis. Feel free to comment below or email me at hockeyspy@gmail.com .
That’s great Kevin! Yeah – I noticed those review at HF as well.
Join me at Twitter:
http://twitter/hockeyspy
or Facebook Group: The Hockey Spy
http://www.facebook.com/s.php?ref=search&sid=3374cfd3a23764b1297ba3d832d92992&init=q&q=kevin%20hopson#/group.php?gid=60905528656&ref=ts
Cheers!
Happy Canada Free Agent Frenzy Day!
Chris,
I think I was quoted a few times in the McKeen’s Draft Guide regarding a couple of American players I covered (Drew Shore and Jeremy Morin). Like I said, I help contribute to some extent but most of my coverage is geared towards prospects already in the system.
When I wrote for Hockey’s Future, we would do annual reviews of each organization’s draft from five years prior. It was a good way to evaluate their draft success and find out who the busts, gems and diamonds in the rough were.
Kevin,
Do you help out with the McKeens’s print issue that comes out with their 150 ranked prospects, as well as prospect rankings for each team?
I really that one!
Thanks Kevin for clarifying further – I definitely see your point.
It will be even more interesting to look back at 3, 4 & 5 yrs intervals and compare the various rankings against how the NHL teams did in their drafting, once we have a better idea of how the prospects are developing.
Cheers,
Chris
I understand what you are saying, Chris. Normally, there is not a significant variance between a service’s mock draft and their actual rankings. However, I was only pointing out that there is in fact a difference in most cases, which would make for a more logical comparison. There were a few off-the-board players taken in the first round (when compared to the consensus top 30) and no one can really know what a particular organization is thinking, especially outside the top 10. Even if you know their drafting tendencies, needs, etc., it can sometimes be a crapshoot. Anyway, good job on the predictions.
Thanks for the comments, Kevin!
I agree with you. I did state that using the rankings of a particular service was a limitation of the analysis.
As I do my own profiles, I do not have the mock drafts put out by the analyzed services. If someone wants to forward those to me or to analyze them theirselves, I would welcome and appreciate that, as well.
I would hope that the mock drafts are at least 2 rounds as comparing 1 round mocks is kind of futile in my opinion.
Another thought on this is that no matter if the ranking service is predicting how a draft is going to actually go down, they are trying to predict and rate whom they feel the best prospects are, in a particular order. Obviously, there are variances in the pro NHL scouting teams, but until proven otherwise, they are the standard as which to judge against. Of course, the track records of some NHL teams would offer ammunition with which you could argue.
I guess, what I’m getting at, is that a mock draft should not be that different from a particular services ratings. I, admittedly had some variances between my rankings, but prospects did fall or rise more than 5 positions in difference if recall.
By the way, I scout for McKeen’s and do not object to your analysis. I was just suggesting an alternative, which would likely be more accurate (could make McKeen’s look worse for all I know). In any event, I do not cover draft prospects that much for McKeen’s so I have very little say in the rankings. My purpose is to evaluate prospects who have already been drafted and are currently playing in NHL farm systems (ECHL & AHL).
Most scouting services (if you pay for them) provide mock drafts and they are different from their respective rankings. As a result, this would be a much more accurate measure for your second test.