This article has nothing and everything to do with Kris Russell. What it’s really about is something very bizarre with the analytics that are being sold to the Oilers, but Kris Russell is the test case here. Yesterday Sportsnet writer Mark Spector published
this piece on how Kris Russell is defying analytics. What I think Spector stumbled onto, though, is that the Oilers are being sold a set of Black Box analysis that is impairing their judgment and obfuscating the truth.
The piece by Spector is certainly indicative of the fact that this was also the writer’s first foray into the advanced stats world. It’s rife with assertions that don’t match current ideas about what the various statistics mean. For example on the topic of PDO and on-ice save percentage, it concludes that Russell’s on-ice .945 save percentage means “the quality of shots Russell has surrendered this season are inferior to virtually every other Oilers defenceman.” This, of course, is not the case at all. The percentage only means that Talbot has stopped 94.5% of the shots he’s faced with Russell on the ice. It has no inherent link at all to scoring chances, which are tracked very separately from save percentage. (Note: Players have an effect on expected goals for and against, but they can also just be getting great goaltending behind them. In the case of Russell all signs point to great goaltending)
But this article isn’t about Spector’s conclusions or personal analysis. What’s the very most interesting and important thing to come from Spec’s article comes from his level of access to the club. This is something the Oilers don’t hand out to non-traditional media (like me or any other established blogger). Even though Edmonton was the early hotbed for analytics, nobody with an active interest in it has actually infiltrated the ranks of the major papers or sports media companies in Edmonton. So we have a pretty thick divide between people who have access to ask questions to the Oilers directly about stats and the people who actually want to ask them.
Happily, the Kris Russell debate piqued Spector’s interest enough to look into it and he asked questions to Peter Chiarelli, Edmonton’s General Manager. The answers raise huge red flags to me and have done so for others who have been following the analytics movement with at least a moderate level of interest.
The first thing we should first note is that, as Spector notes in his article, the Oilers maintain access to a set of numbers that we do not. We have access to publicly available numbers and analysis from places like Naturalstattrick.com, stats.hockeyanalysis.com, and Corsica. The stats largely get their information from the publicly available data generated by the NHL itself. The numbers are testable, the definitions are clear, the information is not altered because a single person doesn’t like the outcomes.
The numbers that the Oilers have access to are what’s referred to as “Black Box” numbers. Some firm has generated these numbers and presenting them with (we assume) some analysis to go along with it. We do not know their process. We do not have access to the numbers. We cannot test the numbers because they are hidden from us. We do not know what kind of analysis is being done, by whom, and for what profit.
Black Box analysis doesn’t mean it is wrong, by the way. Only that we need to take their word for it that they are right. The key to making a living off of this kind of analysis is sales. Someone needs to sell the team on the fact that this analysis is what they want and makes them better or gives them a competitive advantage.
Enter the Oilers, Mark Spector, and Kris Russell.
The Sportsnet hockey writer got the Oilers General manager talking about Russell and he said a few things that are very revealing. On Russell’s ability to pass he said this:
He ranked second (among NHL defencemen) last year in passes leading to (offensive zone) entries. With his skating and effective shot blocking, there are numbers that go the other way from traditional Corsi.
This is passes that lead to an (offensive zone) entry. So he’s got to get the puck, make the play to get into a passing position, then he’s got to make the pass that leads to an offensive zone entry — which is a very significant part of the game when you want own the puck from zone to zone.
He was second in the league behind only Niklas Kronwall last year … ahead of players like Duncan Keith, Kris Letang, Cam Fowler, Ryan Suter, Mark Giordano, Niklas Hjalmarsson and Erik Karlsson.
Now there is a lot to digest here. He’s made part of this assertion before, just after the Oilers signed Russell. Specifically, he’s claimed that Russell was a league leader in passing. The problem is that it runs counter to what other publicly available passing trackers have recorded. Neither the Passing Project run by Ryan Stinson for
years past nor
TheOilersRig’s WheatNOil’s defensive tracking from this season support the idea that Russell was passing with quality or volume to be considered an excellent choice. In fact, he was somewhere between very poor and poor in both of these.
So we have a bit of a problem here in that the Oilers General Manager is being told something and making decisions based on something that just doesn’t hold up to public scrutiny. Now, it’s possible that both projects somehow came to the same incorrect conclusions twice, separately, and with methodology that is open to examination, but it isn’t likely. Based on what Chiarelli has said here, we have reason to raise a red flag.
Importantly, if we run through the information logically, then we should come to a logical result. The claim made by Peter Chiarelli is that Kris Russell is a leader in passes that result in clean and controlled entries to the offensive zone. This is an important skill for any defender because we know there is a relationship between gaining entry to the offensive zone with possession and creating shot attempts, scoring chances, and ultimately goals.
If Kris Russell is a leader in these types of passes then we should see the Oilers receive a boost in these types shooting of events. More zone entries from a player logging big minutes should result in more attempts, chances, and goals. That is the natural and appropriate line being drawn between entries and offense.
Once again, though, none of the available information supports that whatsoever. Even Mark Spector’s piece acknowledges that Edmonton’s shots for drops to its lowest rate when Russell is on the ice. So too does it’s unblocked shot attempts and total shot attempts. Naturalstattrick also records that Edmonton’s Scoring Chances For per 60 minutes are the lowest with Russell on the ice.
So how is it possible for a player who logs huge minutes and who is also their best passer (leading to offensive zone entries) to have a negative effect on the offense? How is it possible for all of those things to be true? Ultimately they cannot be all true. It just doesn’t withstand scrutiny.
Final point I want to make here is about the statement, “He was second in the league behind only Niklas Kronwall last year … ahead of players like Duncan Keith, Kris Letang, Cam Fowler, Ryan Suter, Mark Giordano, Niklas Hjalmarsson and Erik Karlsson.”
Analysis, and especially statistical analysis for sport, has to make sense. It can have surprising results but it cannot produce nonsense. If Peter Chiarelli was told by some analytics firm that Niklas Kronwall and Kris Russell ranked 1 & 2 respectively in passing that lead to zone entries and they were ahead of Keith, Letang, Suter, Giordano, Hjalmarsson, and Karlsson then once again red flags should have gone off. If a statistic you value because it leads to offense rates poor players ahead of great ones consistently then there is a problem along the line somewhere.
For example, 5v5 Points per 60 has some surprises like Benoit Pouliot being rated pretty high over the last 5 years. But also rates Crosby, Benn, Malkin, and Seguin at the top as well. These numbers have to make sense. If the stat rated Pouliot at the top and the stars of the league were at the bottom then we would have major questions about its usefulness. That’s why it is incredibly jarring to see that Chiarelli quote about Russell being ahead of Karlsson.
Taken all together I think Mark Spector’s piece on Kris Russell is a very important piece of journalism. I think it’s important because it shows, I think definitively, that the Oilers are being conned by some unnamed stats firm that is taking them for a ride. They are being given information and analysis that doesn’t hold up to data that is available nor to basic logic.
The Oilers are making decisions and placing priorities on different players based on this information and we should be terrified about it. This article by Spector begs the question of how long they have been getting this information, who is giving it to them, and how much weight are they placing into it? Analytics are supposed to be a way of gaining a competitive advantage, but they are only good if they are pointing you in the right direction. It appears Edmonton is being duped by their stats people right now. It’s only a matter of time before this level of misinformation results in a poor signing, trade, or roster management.
Follow me on Twitter @Archaeologuy 
Edmonton Oilers Blogger • 
Join the Discussion:
»