THE EYE TEST IS FAILING (WHEN DRAFTING CHL FORWARDS IN THE 1ST ROUND)

For evaluating the NHL Entry Draft, I use a combination of statistical based analysis alongside scouting reports provided by the various draft guides available. Both methods of analysis provide insights into a prospect, but that does not mean that both are of equal value. Therefore, how can we test which is a better method? For this purpose, I have chosen to test which method is better for drafting CHL forwards in the top 30 selections. To do this, I evaluated the average amount of NHL games played, points and points per game for CHL forwards drafted in the top 30, between 1990 and 2010. Only players that were in their first eligible draft season were included. The results are:

For comparison, what if the players in each draft year were selected based on their points per game in their draft season (regular season + playoffs). Therefore, I have re-ordered each draft year on this basis. For example the 1990 draft order for CHL forwards selected in the top 30 is:

The re-ordered draft order based on points per game is:

I repeated this for each draft year and re-calculated the average amount of NHL games played, points and points per game for each draft selection between 1 and 30. The results are:

Therefore, to see which of the two lists provide better results, I evaluated the R² between draft selection and games played, draft selection and points, and draft selection and points per game. The results are:

With draft selection and games played, there is little difference between the R² using the actual results of the drafts (0.25) versus the re-ordered version, based on points-per-game (0.28).

However, when draft selection and points are evaluated between the two lists, the results are very different. The R² is nearly double for the re-ordered version, based on points-per-game (0.54) rather than the actual results of the draft (0.30). Therefore, while both methods are similar in explaining the amount of games played, it is not when it comes to points.

The difference is even greater when comparing draft position against average points-per-game, as the R² based on the actual draft results is 0.16 meanwhile it is 0.57 for the re-ordered list.

Therefore, for over two decades, scouts have clearly not put enough weight into what the stats are telling them and chose rather to rely more so on the eye test evaluation. When the first round is re-ordered, by points-per-game, the fact that the R² improves considerably, when evaluating average points and average points-per-game against draft position, suggests that teams are selecting forwards in the incorrect order. There are multiple reasons why teams would be making mistakes in their evaluation, with one reason being putting too much emphasis of their evaluation based on only a small sample of games that were scouted. By doing so, teams put more weight on those games that they attend and place less weight on the multiple games that they do not. In addition, scouts may be putting too much weight on the wrong skill attributes such as size, toughness, defensive play and leadership and not enough weight on more important aspects such as hockey sense and individual skills such as skating, shot, ability to give and receive a pass, creativity, etc. Another possible reason is that teams are being conservative and selecting a forward perceived to be the safer pick. In many cases, this leads to teams selecting a bottom 6 role player while passing on a player with top 6 potential. Therefore, to improve upon their drafting success of first round CHL forwards, teams must continue to move towards a more statistical based analysis away from the old method of using simply the eye test. Simply put, the eyes are failing them.