Modern basketball debates often include one of the parties citing advanced analytics to prove their point. But are those metrics any good?
While some may shy away from numbers when talking about athleticism, others have embraced the statistical revolution. We were curious, though, which of those numbers we should reference in our player evaluations. Is there a catch-all, all-in-one composite metric that has the best reputation and has the most accurate assessment of a player’s holistic impact on winning?
We wanted to find out so, for this story, we surveyed some of the most trusted thinkers in the basketball community.
HoopsHype received answers from nearly 30 participants, including various media members as well as individuals who have a combined experience with more than half of the teams in the NBA. Answers came from folks at every level within an organization, including those who work on a coaching staff as well as several different directors of analytics departments.
Most who answered spoke on the condition of anonymity because they are currently employed for NBA teams and felt that speaking publicly could reveal proprietary information about their teams.
But some like Cory Jez, who was the head of analytics for the Utah Jazz, were kind enough to walk us through the key principles that a team analytics staffer would want to incorporate in their analysis.
“More representative all-in-one metrics need to capture the impact players can have when they don’t log a box score event (see: basically every Rudy Gobert possession ever),” said Jez. “It’s much harder to see the impact a player like De’Anthony Melton has compared to a different substitute like Lou Williams.”
Jez told us his criteria for a good formula would include possession-based data featuring more than just box-score metrics; methods that are Bayesian in nature that consider the individual player contribution and not just possession results; inputs that include tracking data; statistics that properly handle the sample size problem of single-season data by including historical information.
While we received overwhelming interest in this project from individuals like Jez, others declined participation. Some felt that as a whole, catch-all stats are flawed and do not very accurately measure talent or performance.
“I don’t really use any,” said one executive, who is the president of basketball operations for a team in the Eastern Conference. “They are all pretty bad.”
Others were less critical but felt that while all-in-one composite metrics are constantly getting better, the future of analytics is headed away from these measurements altogether.
“If I could add a wrinkle to your story, it would be that all-in-one stats are overused – that the next phase of basketball analytics is all about context-dependent numbers,” said another front office member from the Western Conference. “That would be the most honest quote I could give.”
This executive feels that analytics will move away from ridge-regression-based stats and instead attempt to answer questions about forecast future performance based on roles the player had for their team (e.g. BBall-Index.com and Backpicks.com have metrics on lineup spacing, playmaking value and defensive versatility).
The executive also added that “averaging across a few numbers you trust” (which is something Owen Phillips suggested in this newsletter) is “probably the way to go” to get the best evaluation.
However, the most common feedback to the survey we received was that most teams focus on their own custom-developed systems when evaluating players.
But those measurements aren’t available to the public or to the media and can’t be readily cited. Ultimately, the goal of this project is to provide the most updated feedback on the evaluation tools that you can actually use.
Based on conversations with some of the most trusted names in basketball, here is what we learned about the state-of-the-art, publicly available metrics. The following rankings included below are sorted from least trusted to most trusted.
11
PER
Full Name: Player Efficiency Rating
Metric Type: Box score (not estimating player plus-minus)
Developer: John Hollinger
Data: ESPN.com
Most Recent Leaders: Nikola Jokic, Joel Embiid, Giannis Antetokounmpo, Zion Williamson, Jimmy Butler
Player Efficiency Rating (PER) is an all-in-one metric created by John Hollinger. He originally introduced this statistic for ESPN.com back in 2007, though he had used the metric much earlier than that.
Hollinger created a unique pace-adjusted, per-minute rating of player productivity and it had a home on the Worldwide Leader of Sports. Not only did PER have a prominent placement but it was easy to interpret. The league average score for PER is always 15.0, which makes it digestible and scalable to compare to someone’s points per game.
As a whole, the introduction of PER was a gigantic step forward to welcome more advanced analytics into the mainstream and it is still regularly cited among even casual fans.
Hollinger was a pioneer in his own right and has enjoyed a long career in basketball – he worked in the front office for the Memphis Grizzlies and is now a writer for The Athletic.
Even though the stat has its shortcomings (you can read about them here, here or here), PER represents an important milestone in the history of basketball analytics. Additionally, due to the fact that PER’s limitations are very well-cited, it’s easy to understand the blind spots.
“I did not list PER as a metric that I trust,” one executive, who is currently the director of analytics for a team in the Eastern Conference, told HoopsHype. “But it is ubiquitous and it is easy to understand what it is and what it isn’t. That makes it useful as a quick and dirty summary statistic.”
SURVEY SAYS: Among the 29 individuals who participated in our survey, 22 responded that they did not trust PER (Player Efficiency Rating) as an effective all-in-one metric. None of the participants who answered said PER was their preferred catch-all statistic.
10
WS/48
Full Name: Win Shares per 48 minutes
Metric Type: Box score (not estimating player plus-minus)
Developer: Justin Kubatko
Data: Basketball-Reference.com
Most Recent Leaders: Nikola Jokic, Joel Embiid, Jimmy Butler, Rudy Gobert, Giannis Antetokounmpo
Anyone who read Moneyball (2003) by Michael Lewis is familiar with the concept of win shares in baseball. Originally developed by Bill James for the MLB, the conceptual goal of win shares is to assign credit for team success to individual players.
Using some of the ideas Dean Oliver explained in Basketball on Paper (2004), Basketball-Reference’s Justin Kubatko created a version of the evaluation that translates to the NBA.
It’s a fun measurement because it assigns a score that estimates exactly how many wins a player contributed to his team’s record because after all, getting wins is the ultimate goal of a front office.
But based on results from retrodiction testing by Ben Taylor for Nylon Calculus, Win Shares (much like Box Score Geek’s Wins Produced) yielded particularly volatile results.
SURVEY SAYS: Among the 29 individuals who participated in our survey, 14 responded that they did not trust Win Shares/48 Minutes (WS/48) as an effective all-in-one metric. Only one of the participants who answered said that Win Shares were their preferred catch-all statistic, but curiously that individual was actually the highest-ranking individual with an NBA team of anyone who filled out our questionnaire.
9
FIC
Full Name: Floor Impact Counter
Developer: Chris Reina
Metric Type: Box score (not estimating player plus-minus)
Data: RealGM.com
Most Recent Leaders: Nikola Jokic, Russell Westbrook, Giannis Antetokounmpo, Nikola Vucevic, Luka Doncic
Floor Impact Counter (FIC) is RealGM’s version of PER and PIE. It was developed in 2007 and it differs from other efficiency stats in that it assigns more weight to assists, shot creation as well as offensive rebounding.
It is sometimes presented as FIC40, which is an individual’s FIC per 40 minutes. This metric favors players who can fill up the box score like Russell Westbrook. But anyone who has watched Westbrook over the past few seasons knows that filling up the box score doesn’t always translate to winning.
While it’s not widely cited in the NBA, RealGM has data for other levels of play (such as college basketball, G League and international) so FIC can definitely come in handy in those instances.
SURVEY SAYS: Among the 29 individuals who participated in our survey, eight responded that they did not trust Floor Impact Counter as an effective all-in-one metric. None said FIC was their preferred catch-all statistic.
8
WPA
Full Name: Win Probability Added
Metric Type: Box score (not estimating player plus-minus)
Developer: Mike Beuoy
Data: Inpredictable.com
Most Recent Leaders: Damian Lillard, Stephen Curry, Zion Williamson, Nikola Jokic, Jayson Tatum
Back in 2014, Mike Beuoy introduced win probability contributions, which are weighted to give more credit to clutch shots. He defined some of his methodologies here, here and here.
The most interesting takeaway is that he focuses on missed shots, made shots, turnovers and free throws while ignoring rebounds, assists, blocks and steals. On his site, you can also sort individual performance based on how well someone did in clutch minutes compared to how well they did while in garbage time.
Inpredictable has another catch-all metric named kWPA (kitchen sink win probability added) that factors in all box score stats like rebounds, assists, blocks and steals. They also do great work examining clutch shooting and pace of play in the NBA.
SURVEY SAYS: Among the 29 individuals who participated in our survey, five responded that they did not trust Win Probability Added as an effective all-in-one metric. No participants who answered said that WPA was their preferred catch-all statistic.
7
RPM
Full Name: Real Plus-Minus
Metric Type: Box score using tracking data and on-off impact hybrid (Prior Informed RAPM)
Developer: Jeremias Engelmann, Steve Ilardi
Data: ESPN.com
Most Recent Leaders: Stephen Curry, LeBron James, Rudy Gobert, Paul George, Giannis Antetokounmpo
ESPN’s Real Plus-Minus (RPM), which was introduced in 2014 and is credited as launching a new era of composite metrics, aims to replicate the plus-minus component of a box score while also sifting through the NBA’s play-by-play data.
RPM was next in the lineage that included the groundbreaking WINVAL ratings (developed by Wayne Winston and Jeff Sagarin in 2002), APM (adjusted plus-minus developed by Dan Rosenbaum in 2004) and xRAPM.
Especially with its prominent placement on ESPN, RPM was long considered to be the gold standard among the adjusted plus-minus stats. But then sometime last year, something strange happened.
“RPM created by Jeremias Engelmann is not the one currently displayed on ESPN,” responded one individual, who works as the director of basketball analytics for a team in the Western Conference. “I would rank the former very highly while the latter is in my estimation fairly useless.”
Overall, the opinions about ESPN’s RPM were the most divisive of any metric mentioned. Some even felt that there may be something broken in whatever coding is currently formulating the results.
SURVEY SAYS: Among the 29 individuals who participated in our survey, two said ESPN’s Real Plus-Minus (RPM) was their preferred all-in-one metric. Eight others said they trusted RPM whereas 11 responded that they did not trust RPM.
6
BPM
Full Name: Box Plus-Minus
Metric Type: Box score (estimating player plus-minus)
Developer: Daniel Myers
Data: Basketball-Reference.com
Most Recent Leaders: Nikola Jokic, Giannis Antetokounmpo, Stephen Curry, Jimmy Butler, LeBron James
Box Plus-Minus (BPM) aims to evaluate how much a player contributed when they are on the court. BPM incorporates box score performance, team performance and player position to rate how well a player performed above league average per 100 possessions.
BPM is essentially the same statistic as VORP (Value Over Replacement Player) but the latter accounts for playing time.
For BPM, the league average is set at 0.0 and you can compare individual performances (e.g. 10.0 is considered to be an all-time great season, 8.0 is considered MVP level, 4.0 is someone who deserves All-Star consideration, -2.0 is a bench player, etc.) quite easily.
When he developed BPM 2.0, Daniel Myers wrote that he wanted his formula to only use stats that are readily available.
Unlike ESPN’s RPM, this measurement is comparably simple and doesn’t include tracking data. As such, despite relying on less information, it’s interesting to note that many actually prefer BPM over the more complex RPM.
The results for BPM can be found at Basketball-Reference. But it’s also worth noting that Ben Taylor, author of Thinking Basketball (2016), has his own version of BPM that many said they trusted as well. Those results can be found here.
SURVEY SAYS: Among the 29 individuals who participated in our survey, two said Box Plus-Minus was their preferred all-in-one metric. Eleven others said they trusted BPM as an effective all-in-one metric whereas three said they didn’t trust BPM.
5
RAPM
Full Name: Regularized Adjusted Plus-Minus
Metric Type: Pure on-off impact with no box score
Developer: Joe Sill
Data: NBAShotCharts.com
Most Recent Leaders*: Stephen Curry, Chris Paul, LeBron James, Giannis Antetokounmpo, Rudy Gobert
As mentioned in the section about ESPN’s RPM, Regularized Adjusted Plus-Minus (RAPM) works off the basic principles of Adjusted Plus-Minus (APM), which applies linear algebra to standard plus-minus found in a box score.
But the main difference from APM is that RAPM applies a linear ridge regression filter to account for regularization. You can learn more about the Bayesian process used here and you can read a history lesson about RAPM here.
One of the most important things to know about these calculations, though, is the difference between prior informed and non-prior informed when it comes to RAPM.
The use of “priors” to the plus-minus measurements like RAPM allows for certain metrics to pass the “sniff test” at first glance. Utah’s former head of analytics, Jez, offered a simple explanation:
“We know that LeBron James is really really good, we shouldn’t have to wait for hundreds of possessions to tell us that,” Jez told HoopsHype. “Instead, modelers can use a prior to start the model off in the right direction.”
Of course, explained Jez, how those priors are created is subjective and it’s what makes the “secret sauce” behind stronger predictive metrics.
There are some obvious flaws with RAPM when it is used by itself, chief among them being that there are no real “basketball” values like rebounds or assists used in the input beyond just the score of the game when an individual is and isn’t on the floor.
Even though RAPM ignores the traditional box score elements, it can be very helpful as a base component in other popular composite metrics like EPM (more on that later!) to measure on-off factors.
It’s also worth noting that several who responded also told HoopsHype that this metric is at its most useful when studying multi-year results (e.g. three-year RAPM and five-year RAPM, which software engineer Ryan Davis updates here) but that single-season data can be misleading.
“It needs to be considered in [the] context of the player and their role,” said one director of analytics for a team in the West. “But it is a good objective measure when you have multiple seasons.”
SURVEY SAYS: Among the 29 individuals who participated in our survey, more than half (15) said they trust Regularized Adjusted-Plus Minus (RAPM) as an all-in-one metric and only two said they did not. But none said that RAPM was their preferred catch-all metric.
* = Leaders based on 5-year RAPM
4
RAPTOR
Full Name: Robust Algorithm (using) Player Tracking (and) On/Off Ratings
Metric Type: Box score with tracking data and plus-minus hybrid (estimating RAPM)
Developer: Jay Boice, Neil Paine and Nate Silver
Data: FiveThirtyEight.com
Most Recent Leaders: Nikola Jokic, Joel Embiid, Rudy Gobert, Kawhi Leonard, Mike Conley
Perhaps the most household name in the data world is Nate Silver, so it’s no surprise to see that his publication FiveThirtyEight has a widely-cited catch-all basketball metric: RAPTOR.
FiveThirtyEight’s previous basketball metrics (Elo and CARM-Elo) relied on BPM and RPM for calculations. But RAPTOR, introduced in 2019, offers original insight from their data scientists. This evaluation incorporates a blend of box score data, play-to-play data, player-tracking data and an on-off component.
The main thing to know about RAPTOR is that it includes a ton of descriptive data that are as specific as fast-break starts, isolation turnovers and distance traveled for perimeter defenders.
One executive told HoopsHype that they preferred RAPTOR because it is created in such a different way than other metrics and that they consider it to be useful as a “robustness” check.
RAPTOR is deservedly well-respected. However, studies have shown that its limitations are that is not as useful as a predictive tool when there is a smaller sample size of data (minutes played) or when there is a significant amount of year-over-year roster turnover.
SURVEY SAYS: Among the 29 individuals who participated in our survey, six said that RAPTOR was their preferred catch-all metric, which made it the third-most-popular in that regard. Eight others said that they trust RAPTOR as an all-in-one metric but an additional seven said that they did not.
3
LEBRON
Full Name: Luck-adjusted player Estimate using a Box prior Regularized ON-off
Metric Type: Box score and on-off impact hybrid (Prior Informed RAPM)
Developer: Krishna Narsu, Tim/Cranjis McBasketball
Data: BBall-Index.com
Most Recent Leaders: Nikola Jokic, Rudy Gobert, Giannis Antetokounmpo, Jimmy Butler, Joel Embiid
One of the newer metrics that has become increasingly popular over the past few years is BBall-Index.com’s LEBRON. While it’s fresh to the scene, it has performed pretty well thus far.
This rating uses box score results and on-off calculations (specifically, luck-adjusted RAPM) for an impact score measured per 100 possessions. The box score component of LEBRON uses weights from PIPM (Player Impact Plus-Minus).
PIPM – no longer publicly available – was created by Jacob Goldstein, who worked for BBall-Index before he was hired to work for the Washington Wizards, Washington Mystics and Capital City Go-Go.
For stabilization purposes, they incorporate their offensive archetypes, which labels players based on the role they play for their team. According to the site, LEBRON is the only impact statistic that measures “role-adjusted, stabilized and luck-adjusted values utilizing actual RAPM calculations.”
BBall-Index also mentions that the “next generation” of LEBRON will incorporate tracking data.
SURVEY SAYS: Among the 29 individuals who participated in our survey, four said that LEBRON was their preferred catch-all metric. Fourteen others said that they trust LEBRON as an all-in-one metric while two said that they did not.
2
EPM
Full Name: Estimated Plus-Minus
Metric Type: Box score and on-off impact (Prior Informed RAPM)
Developer: Taylor Snarr
Data: Dunksandthrees.com
Most Recent Leaders: Nikola Jokic, Rudy Gobert, Stephen Curry, LeBron James, Kawhi Leonard
One unsurprising trend we can pick up from this study is that newer metrics tend to be trusted more than the older ones and the popularity of Estimated Plus-Minus (EPM) is a clear example.
EPM was developed by data scientist and former Utah Jazz basketball analytics coordinator Taylor Snarr. When he introduced EPM back in February 2020, he described it as the “most accurate” all-in-one NBA player metric.
Science proves this as well. Based on a comparison study, EPM did the best job at assigning credit for impact to the correct players. You can see the results here, although it is worth noting that the results of the metric comparison are actually very similar to answers provided in this survey.
Here is what a current Eastern Conference analytics staffer, who felt that the study was set up pretty well and points to how well EPM compares to other metrics, told HoopsHype:
“I will say the part I like most about EPM is that he put together a study that shows how well it projects compared to other metrics. I always wonder why more people don’t do this. If your metric is better, show me why it is, don’t just tell me it is.”
The analyst added that all-in-one metrics that typically perform the best are adjusted plus-minus metrics that include priors.
Meanwhile, like the aforementioned RPM and RAPTOR, Snarr’s EPM is a hybrid model that uses player-tracking data as part of its evaluation.
We asked Steve Ilardi, co-creator of ESPN’s Real Plus-Minus who has worked as a consultant for NBA teams including the Phoenix Suns, what he thought about the metric.
“[EPM] does a superb job of reducing the high noise level inherent in basic RAPM modeling estimates by integrating Bayesian priors derived not just from box score stats (a la BPM) but also from player tracking metrics,” said Ilardi, during his recent conversation with HoopsHype. “[EPM] has gone one step beyond the RPM estimates that Jerry Engelmann and I developed for ESPN, as we began that project without access to player tracking metrics.”
“I regard EPM as the obvious gold standard of all-in-one metrics,” added Ilardi.
Additionally, not only is the data trusted but the website also looks pretty modern. That’s a nice change of pace because some of the other sites that host these metrics are outdated and, admittedly, can be really hard to look at it.
Dunksandthrees.com also offers a great user experience that includes fantastic data visualizations and color-coded efficiency ratings for skills like defense and ball handling.
SURVEY SAYS: Among the 29 individuals who participated in our survey, six said that EPM was their preferred catch-all metric. That was the second-most among all metrics. Eleven others said that they trust EPM as an all-in-one metric while only one said that they did not.
1
DPM
Full Name: Daily Plus-Minus
Metric Type: Predictive tool estimating plus-minus
Developer: Kostya Medvedovsky
Data: DARKO.app
Most Recent Leaders: Giannis Antetokounmpo, Kawhi Leonard, LeBron James, Joel Embiid, Kevin Durant
The “winner” of our composite metric survey goes to DARKO. This is an application developed by Kostya Medvedovsky and hosted by Andrew Patton.
Their website defines DARKO as a “machine learning-driven basketball player box-score score projection system” similar to MLB’s PECOTA or ZIPS. Their application updates daily for every player in the NBA. The tool has inputs from NBA.com, Basketball-Reference and PBPStats.com.
DARKO also accounts for time series and sample size using the complex methods of “exponential decay” and “Kalman filters” to take all historical data into account.
“This is a huge breakthrough because when using possession-level data, it can be tricky to tease out the signal from the noise and oftentimes one season of data is not enough to ensure the proper accounting,” explained Jez, the former head of analytics for the Jazz. “Medvedovsky has solved for this with DPM, allowing him to confidently say when a players’ improvement is more signal than noise.”
The results are a Bayesian model that predicts all elements of a box score. The app shows a projected skill curve for each player highlighting elements such as perimeter shooting and other box-score-specific elements.
For this exercise, in particular, we focused on its results for Daily Plus Minus 2.0 (DPM), DARKO’s version of a catch-all metric. The main difference between DPM and the other metrics is that it solely looks forward.
It won’t answer who should have won MVP but it’s a powerful tool for projections. In fact, DPM beats all the other public-facing metrics in predictive power, based on root mean square error (RMSE). The next two best metrics graded out, in order, were EPM and LEBRON – just like in our survey.
While it might not have the same popularity among casual fans as PER, it might be time to add DARKO’s Daily Plus-Minus to your list of metrics that you check if you’re interested at all in NBA player evaluations – especially because like Dunksandthrees.com, DARKO’s application also has some great data visualization tools.
As others begin to consider what to include in their composite metrics, it is also worth noting that a predictive model fared better than more traditional evaluations.
SURVEY SAYS: Among the 29 individuals who participated in our survey, eight (8) said that DPM was their preferred catch-all metric. That was the most among all metrics. Ten (10) others said that they trust DPM as an all-in-one metric while only one (1) said that they did not.
HONORABLE MENTIONS (stats that are not publicly available)
APM (Adjusted Plus-Minus // Developer: Wayne Winston and Jeff Sagarin)
SPM (Statistical Plus-Minus // Developer: Dan Rosenbaum)
WARP (Wins Above Replacement Player // Developer: Kevin Pelton)
IPV (Individual Player Value // Developer: Talking Practice)
DRE (Daily RAPM Estimate // Developer: Kevin Ferrigan)
SPR (Simple Player Rating // Developer: Nathan Walker)
PIPM (Player Impact Plus-Minus // Developer: Jacob Goldstein)
AuPM (Augmented Box Plus-Minus // Developer: Ben Taylor)
Main Image: Coley Cleary / USA TODAY Sports Media Group
