How the Chiefs Won the Super Bowl with a Bit of Analytics (and a Lot of Org Effectiveness)
Super Bowl 58 was a case study in organizational effectiveness, even though the difference between going first or second in OT is probably a wash
The turning point in Super Bowl 58 was not a football play, but a leadership decision. At the start of overtime, basically the entire United States scrambled to learn the new rules which were adopted by the NFL in 2022 but hadn’t been tested until now: both teams would get a chance to possess the ball, with sudden-death beginning in the third possession if the game was still tied.
The 49ers won the overtime coin toss and elected to receive the ball first - a choice that has been the default for decades where, maddeningly (pun intended), a coin flip has had an outsized impact on the outcome of games. But now the rules have been improved to give both teams a fair shot, as NFL analytics guru Mike McRoberts would explain on NBC following the game:
"I thought it was almost a toss-up," McRoberts explained. "Gun to my head, I would have done what Kyle (Shanahan) did. Whoever designed this rule did a good job, because the outcome should not be determined by the coin flip, and each team gets at least one full possession. It’s not gimmicky at all."
Contra McRoberts, 49ers head coach Kyle Shanahan explained his choice to take the ball first as being influenced by analytics:
“We went through all of the analytics,” Shanahan said. “We wanted the ball third. If both teams matched and scored, we wanted to be the ones who had a chance to go win.”
Hindsight bias is a hell of a drug, so the 49ers coaching staff has come under a lot of fire for not electing to give the Chiefs the ball first. The critiques boil down to a few critical factors:
The team that goes second has an information advantage: Depending on what happens in the first possession, the second team could win with either a field goal or touchdown
The second team has the “gotta-have-it” advantage, and will go for it on 4th downs when necessary.
Finally, how can you use analytics to educate choices for a situation (OT under the new NFL rules) that has literally never happened - that we lack past data for?
Putting Numbers into Context
If I hear the phrase “what do the analytics say?” on a sports broadcast one more time, I might lose it. Sure, as I wrote about previously, analytics has changed the way football is played. That said, the key challenge for decision makers is: because they have access to a wealth of context in terms of what’s really happening on the field, how can one combine multiple sources of information in order to make the best possible decision? There’s a lot of daylight between the two extremes of blindly following data and ignoring it altogether.
Chiefs head coach Andy Reid is a great example of someone who has embraced the analytics revolution, being more aggressive on 4th down decisions for example, but uses his judgement on the field as well. Reid credited Chiefs Analytics Coordinator Mike Frazier for the overtime strategy of electing to receive the ball second, if possible - citing the information advantage (factor #1 above) as one reason.
But the question remains - who is actually right? Did the Chiefs really make the better decision, analytically speaking, or did the 49ers actually make the right call and it’s just that the Chiefs played better?
Putting Context into Numbers
There is a paper “A Markov chain analysis of NFL overtime rules”, and I think the authors deserve a lot of kudos, not just because the research is cool but also because back in 2018 they proposed and simulated a set of “alternative” overtime rules that essentially became the new official OT rules in 2022. The authors used historical overtime data to create markov chain simulations of overtime games, and demonstrated that prior rules heavily favored the winner of the coin toss. However, even their proposed rule changes suggested a win probability of around 55% for the team going first (one wonders if these were the analytics the 49ers were looking at).
The authors addressed aforementioned critical factor #3 by illustrating that you can use simulations to predict what could happen, given a new set of rules, by using older data and making some basic assumptions. The researchers didn’t consider critical factors #1 and #2 though, so to better understand whether team A or team B has an advantage I went ahead and tried a simulation myself, using the same data from stathead.com but making a couple adjustments:
I considered the possibility that if team B receives the ball down 7, and they score a touchdown as well, that they would go for 2 to try and win the game. The historical average conversion rate for 2pt conversions is around 48%.
I added a +50% “gotta-have-it” multiplier for team B’s likelihood of scoring a touchdown or a field goal under situations where they would be going for it on 4th down. For example, where the probability a drive ending with a field goal was 32%, I adjusted this to be 48%. This adjustment is admittedly a guess, but was educated by historical data which indicate that if punting is not an option, drives ending in a score are about 50% more likely.
Here is a visualization of the markov chain, where the starting point is ‘A_First_Pos’, and the game ends if A_Wins or B_Wins is reached:
If we actually simulate this by taking a “random walk” through the chain, possession by possession, we find that by the end Team B has a better chance (49%) than the 2018 research suggested, (nearly a perfect 50/50!) but not necessarily an advantage.
Translating Organizational Agility to Competitive Advantage
Because it doesn’t seem like either team’s overtime strategy led to a significant advantage in either direction, I want to step back from the numbers and zoom out for a bit. What was astonishing about the whole OT rules situation is that it gave us a window into the organizational effectiveness gap between the two teams. After the game, multiple 49ers players mentioned their lack of familiarity with the new OT rules. In stark contrast, Travis Kelce recalled the Chiefs’ extensive preparation on his podcast New Heights:
"Every single week, we talked about overtime rules in the playoffs. Fraz (Mike Frazier) was up front giving what we would do in all these scenarios. So, everybody on the team, like we knew exactly what the best situation was and how we were going to handle it, how we were going to attack it…"
Even if the actual decision of whether to go first or second in overtime didn’t move the needle (and it’s possible it did - we don’t know what the Chiefs did with analytics behind the scenes), it had to have been empowering for the Chiefs players to have been so well prepared. In the biggest moment on the biggest stage, the 49ers were trying to learn and adjust to the rules with the rest of us in real time. The Chiefs were able to lock in and focus on their game.
There is an interesting thread of research on organizational agility, the ability for organizations to respond and adapt to external changes. One contributing factor that is highlighted in the research (ex. Felipe et al. 2016) is absorptive capacity, referring to an organization’s ability to take in novel, external information and effectively put it to use. If this is the difference between winning and losing in football - a game with explicit, finite parameters - just imagine what’s being left on the table in your line of work.