Bracketology: Analyzing impact of metrics in NCAA Tournament selection, seeding

Bracketology season is on the horizon. Which metrics warrant major attention and how does the committee evaluate the different types?

The 2021-22 college basketball season is speeding along. With fewer than 10 weeks until Selection Sunday, team sheets are soon to be flooding conversations. Bracketology is far from hard science, but the process of determining which teams might be worthy of NCAA Tournament inclusion has become more convoluted (in a good way) in recent years. 

The introduction of the NCAA Evaluation Tool (NET) is the most well-documented change made by the selection committee, but it is not the only one. In fact, there are six core metrics, including the NET, that appear on official team sheets. The NET determines quadrant records and is thus the most important, but the committee also dives into resume and quality metrics. 

The Kevin Pauga Index (KPI) and Strength of Record (SOR) are deemed the “resume” metrics. These measure the difficulty of a team reaching its record, based on the end results of games — rather than how those results came to be. “Quality” metrics generally focus on team efficiency (i.e., points per possession) to assess the caliber of performances. BPI, KenPom, and Sagarin are the quality metrics.

Juggling the value of resume vs. quality metrics is one of the more difficult bracketology discussions. Of course, boasting high rankings in both departments is the ultimate goal heading into March, but that isn’t feasible for every program. By the time the selection committee meets each year, there are teams that are held in much higher regard by one or the other.

With the 2021-22 college basketball season at midseason, let’s dive deeper into how the committee appears to address this dichotomy and evaluate which teams are on each side of the fence at this point. 

Ten mid-majors eyeing at-large March Madness bids
—Ranking most likely first-time NCAA Tournament teams
—DPI: Game Predictions | Team Grades | Player Rankings

The selection committee’s approach to date

Conventional wisdom indicates that quality and resume metrics would eventually level out over the course of a season. With each game played, a larger sample size exists to draw the two closer. A team of a given quality should acquire a resume of similar stature. However, that isn’t always the case.

There are programs every year that enter Selection Sunday with sizably better rankings in one category versus the other. There are currently two years’ worth of data during the “NET Era” to evaluate how the committee addresses this debate. 

The committee’s approach can be assessed — at least to some extent — using a model that projects NCAA Tournament seeds by averaging each team’s resume and quality metrics and weighting them evenly.

Below is a look at the teams that have been most overseeded and underseeded over the past two tournaments relative to the model’s placement. The rightmost column shows how the team ranked in quality metrics vs. resume metrics, which may provide a potential explanation for why they were seeded as such.

As it turns out, simply averaging resume and quality metrics and giving each equal weighting actually resulted in very similar seeding to what the selection committee did. That model projected 80 of 92 at-large worthy teams either perfectly or within one seed line – that is a pretty high rate. (Note: The term “at-large worthy” here refers to any teams seeded above the bubble cut line.)

However, of the teams that were underseeded relative to the model by two or more seed lines, five of the seven were rated better by resume metrics than in quality ones. Conversely, most of the teams overseeded benefitted from stronger quality ratings (aside from West Virginia). A preference for the KenPoms of the world appears to be in the committee’s brain.

Overall seed lines aren’t necessarily the most important part of bracketology, though. Selecting the correct field of team is obviously another crucial piece of the puzzle. So, do these same takeaways apply when looking at the bubble?

Evaluating how the bubble is addressed

The short answer is no. In fact, while the overarching theme of “overseeded” at-large teams is having better quality metrics, that is not the case as the committee progresses through the seed lines. The debate is treated essentially evenly through seed lines No. 6-8, with the average rankings matching up with one another.

However, from No. 9 seeds on – as well as the teams deemed to “First Four Out” – there is a significant shift towards favoring resume rankings. The change rewards the teams that compiled the strongest resumes, even if their efficiency numbers are not as high as some others under consideration.

This makes sense as the committee selects the teams that make the field before actually seeding them. As the seed lines draw closer to the cutline, it becomes more clear the teams that were selected based primarily on resumes.

A gap appears as the seed lines approach the bubble. This gap indicates the preference towards resume metrics at that time. (Note: “F4O” = First Four Out)

When evaluating teams right on the bubble, No. 11 seeds and First Four teams are nearly identical when it comes to average quality ranks (plus-1.14 difference as compared to resume metrics). However, teams that actually made the tournament have been an average of 4.3 spots higher in resume rankings.

It is also important to look at the biggest extremes on the bubble.

The 2021 Wichita State team had the largest difference between quality and resume metrics of any at-large over the past two tournaments. The Shockers danced despite averaging just a 79.0 ranking in quality metrics. They were 41.5 spots higher in their resume metrics compared to quality measures. The 2019 versions of Temple (plus-28.0 in resume metrics) and St. John’s (plus-24.2) similarly made the field while being several spots higher in measurements based on results rather than efficiency.

In total, eight bubble teams (No. 10 seeds or lower) have made the last two NCAA Tournaments while being more than 10 spots higher in resume rankings compared to quality rankings. Only three such teams have done so while more than 10 places higher in quality measures.

While not a hard-and-fast rule by any stretch, it appears as though the committee has been favoring resume metrics over quality metrics over the past couple of seasons when it comes to the bubble.

How is the 2021-22 season shaking out?

Diving into this season, in particular, there are several teams that already emerging as leaning towards one side of the quality vs. resume spectrum. Based on the conclusions above, it is better to be on the quality side when already seen as a “lock” to reach the Big Dance. On the other hand, having strong resume metrics is more beneficial while flirting with the bubble. 

Here is a deeper dive into which potential at-large teams have the widest gaps in resume and quality at the midpoint mark of the 2021-22 college basketball season. Potential at-large teams are any that have either an average quality ranking or an average resume ranking in the top 100. As of Jan. 5, there were 112 teams that qualified as potential at-large selections.

Teams dependent on quality rankings

It should come as no surprise to learn that high-majors dominate the list of teams most dependent on quality rankings. When glancing over the list of programs in this chart, there is an overarching theme: strong preseason quality metrics. All 20 of these teams ranked in the preseason KenPom Top 85 (15 in the Top 45). 

While not every team here has been disappointing relative to expectations, a whole lot of them have been. Michigan, Oregon, Florida State, Virginia Tech, Arkansas and Memphis would all make lists for the most disappointing teams in the country to this point – at least from a resume standpoint. That idea is reflected here as well.

Maintaining those high preseason quality rankings is crucial to these teams’ postseason applications. High-majors are not only more privy to higher preseason rankings but also have the ability to schedule differently. Most notably, performing extremely well in buy games against worse competition can bolster or maintain quality ranks. 

For reference, here are the KenPom nonconference SOS ratings for several of the teams above: Northwestern (No. 355); Texas Tech (No. 354); Iowa (No. 344); St. John’s (No. 339); Kansas State (No. 320); Kentucky (No. 317); Indiana (No. 301); Washington State (No. 290); and Arkansas (No. 287).

Additionally, losing in close games to good opponents helps teams maintain a solid quality ranking. Drake is a good example. The Bulldogs lost games to Belmont, Alabama, North Texas, and Missouri State – all KenPom Top 100 teams – by an average of 5.5 points. Their largest margin of defeat (10 points) came against Clemson in overtime.

In short, landing among the quality-dependent teams appears to a benefit of strong preseason quality metrics, dominant performances in buy games, losing close games against strong opposition — or a combination of all three.

Teams dependent on resume rankings

As one would expect, the opposite takeaways come when evaluating teams with better resume rankings. There are far more mid-majors in this chart, many of whom will likely be written about as “potential Cinderella teams” later in the season. New Mexico State and Iona are right near the top of the list as two teams massively outperforming low quality rankings. The Gaels, in particular, are still working to overcome exceedingly low preseason quality metrics.

Colorado State, San Francisco and Belmont are all teams that have received a fair amount of press this season for being among the nation’s top mid-majors. The Rams are even nationally ranked right now. Yet, their resume metrics are miles ahead of their quality rankings to the point that they cracked this chart. That goes to show that perhaps they still aren’t receiving enough attention for the actual results that they have put forth to date.

Not all of the teams in this chart are going to be able to compete for at-large bids this season. There are already a few (Princeton, Navy, Grand Canyon, etc.) that would have a very difficult time pulling off that feat. However, there are several others that could find themselves right in the thick of that conversation come March.

With the ACC and Pac-12, most notably, struggling as high-major conferences, the door is open for more middies to swoop into the NCAA Tournament. This article already discussed the value of having strong resume metrics when it comes down to near the cutline. That is especially great news for teams like Iona, Ohio, Wyoming, San Francisco and Belmont.

Overall analysis of the debate

The resume vs. quality rankings debate is not going to slow down anytime soon. There are only two seasons of data of the committee using these six metrics, and their application is sure to change over time. Still, adding KenPom and SOR – among others – to the team sheets has made the selection and seeding process more advanced than it was in the past. 

Is the Big Dance about selecting “the best teams” or is it about selecting “the teams that earned it most” in the regular season? That question is at the crux of the debate. Based on the last two NCAA Tournaments, the selection process relies more heavily on resume while seed placement is more dependent on quality.

If that continues to be the trend, then it could lend more clarity on how to evaluate bracketology conversations over the final month of the regular season. That knowledge could also affect how teams put together their nonconference schedules. For now, though, these two very different types of metrics make for a tricky juggling act when trying to curate college basketball’s biggest stage.



Categories: Bracketology, Words