Current Discussions subPWE's Stats PWE NBA Bracket 2015 Home
How Hollinger's draft rater is failing stat-heads - Pinwheel Empire

How Hollinger’s draft rater is failing stat-heads

submitted 6 years ago by in Stats

I am a pretty avid defender of Hollinger's PER stat. It's good because it can be calculated with nothing more than basic arithmetic (the formula is just long, the computations themselves are simple), it's robust so players generally have the same PER year to year, it's good at predicting team performance, and it's got a strong theoretical basis (points are good, turnovers are bad, misses are kind of bad, etc.). Sadly, it's for exactly these reasons that I've soured on his draft rater. After five years of putting them out, it's become increasingly apparent to me that the draft rater is a bunch of numerical gobbeldy gook surrounded by Hollinger's always rather insightful writing.

A regression is no substitute for theory

When someone sucks at statistics, the first thing they do is run a regression to solve every problem, then shoot everybody down by saying "I ran a regression!" The problem is, the social science garbage bin is full of models that perfectly fit past data but couldn't predict for shit. To me that's a big chunk of the draft rater.

Being able to run a 16 variable regression is neat, I suppose. But when you end up rating steals as a massively important indicator of NBA success, a good statistician needs to do a little more digging. Especially in a situation like this, since in a global sense Hollinger is actually dealing in very small samples (he only needs to hit the top 5 players in each draft to consider it a success, and only has data for the last 10-12 years). This isn't exactly the history of every baseball at bat, where a correlation will have enough weight to be entitled to a presumption of truth.

To make matters worse, Hollinger appears to have taken his pent-up desire to fiddle with PER and applied it to the draft rater. While it's good to keep incorporating new information into a statistical model, tweaking it significantly every year is massively problematic. It creates confusion about how prospects compare to one another (and whether that is a comparison we can make at all). It also makes any sort of validation of the productivity of the model impossible since it changes from year to year. Models should be created to be tested, but all we can evaluate with the Draft Rater is its creator's whims.

Telling a story

The methodology of the Draft Rater is one thing (and it's a secret, so it's hard to criticize). A different basket of issues altogether is how Hollinger presents his findings in the column he writes. The biggest issue here in the past was that he used the data the regression was based on to prove how great it was. He talked about how the Draft Rater identified CP3, Millsap and Boozer as better players than where they were drafted, but of course they were! The regression was built to account for CP3, Millsap and Boozer's successes!

Though he's moved away from this type of commentary in recent years, the fact that he did it in the first place and continues to use the same rating scale invites people to continue to use this argument on the Rater's behalf. And as mentioned above, it's not at all clear how similar the 2012 Rater is to the one that love Chris Paul so much!

Finally, Hollinger's commentary in defense of a few of the Rater's mistakes appears to me to wipe away a lot of its usefulness:

"So let's take a closer look at two relatively short, stocky power forwards who are among the next names on the list: Sullinger and Draymond Green. Will these guys put up numbers? Very likely. Will they be able to guard their position? That is a much more open question and why they won't go as highly as Draft Rater places them. Green, in particular, is a massive defensive question mark."

That would be OK, except presumably Hollinger already did a million different ad hoc adjustments to weed out guys like that who wouldn't translate, otherwise we'd just use college PER and not something that adjusts for age, height, rebounding at particular heights, an adjustment for "perimeter" vs. "interior." So now he's basically doubly adjusting against players who are too short or too slow.

That'd be fine if he hadn't built the credibility of the model based on how much it loves guys like Boozer, Millsap and DeJuan Blair. It's fine to temper expectations for how well a model will work short of perfection. It's another to equivocate what you've put out as the greatest selling points.

While Hollinger's model is bad and the column he writes about it is worse, it continues to be the only accessible stats-oriented draft analysis anywhere. So I'll continue to read it and rely on it. But if someone is smart and has time, there's a niche for a draft model that is public and more responsibly created and advertised than Hollinger's. That is, until you inevitably get hired by and NBA team and aren't allowed to publish it anymore.