The Science-Practice Gap: Why is Conventional Wisdom on Hiring so Misguided?
Selection decisions are widely viewed as “best guesses” and “judgment calls” despite the availability of evidence-based, data-driven approaches
It’s another week on the app formerly known as Twitter, and that means another squabble between billionaires. I’m going to sidestep the DEI argument, because one of Mark Cuban’s tweets provides a window into something I think is much more interesting: the gap between how hiring methods and decisions are broadly perceived, compared to what the science of personnel selection would suggest.
There is a lot to unpack here. I’m going to make a couple of assumptions - first that Mark’s view of the hiring decision as an “educated guess” represents the median viewpoint. He’s not wrong about the vast majority of hiring decisions being subjective, educated guesses in practice. However, we have over a century of scientific research that provides a clear roadmap to leverage quantitative measures for picking the best candidate. To the degree that hiring decisions are based on a hiring manager’s guess/belief today, this represents a significant missed opportunity, not how things are destined to be.
My second assumption is that when we’re talking about managers hiring who they believe to be the best, or making educated guesses on who is the best candidate, what we are talking about is the hiring method most people default to: unstructured interviews. Let’s quickly define our terms:
Unstructured interviews: are spontaneous conversations where interviewers can ask questions as they come to mind, and then make an overall impression of the candidate
Structured interviews: use standardized, predetermined questions for all candidates as well as formal scoring systems to evaluate a candidate’s responses to the questions
If they didn’t know already, folks like Mark Cuban might be horrified to learn that from a DEI standpoint - an unstructured interview (i.e., letting hiring managers ask whichever questions they want, and making a decision based on an overall impression) is the worst possible method (ok, well maybe brain teaser questions are worse). Without structured criteria, people hire based on judgment alone, meaning that first impressions, charisma, voice and height, and cultural similarity all sneak their way into the hiring criteria. Unstructured interviews are anathema to diversity. They’re also poor from a performance perspective, as we’ll see:
Hiring doesn’t have to be a complete guess
We can boil hiring decisions down to an essential question: Given two qualified candidates, what is the probability we can successfully choose the better one?
Sackett et al. (2021) provide the best existing data on the predictive validity of hiring methods, aggregated from a plethora of studies. They estimate the correlation between unstructured interviews and eventual job performance to be .19, whereas structured interviews offer a significant improvement at .42.
We can convert these abstract figures into the percent concordant (PC) statistic which directly answers our essential question: Conducting an unstructured interview gives us a 56% chance that we would identify the better candidate (an educated coin flip, as Mark suggests), but a structured interview gives us a 64% chance, a marked improvement. Combined with other validated selection methods such as psychometric assessments or work sample tests (my personal favorite), a PC around 70% (correlation = .60) is achievable.
To be charitable to Mark, this means that hiring decisions are still a bit of a guessing game. As Daniel Kahneman and company point out in their book Noise, we are all subject to “objective ignorance” of the future - it is simply impossible to foresee all of the factors that contribute to someone’s future performance on the job. But while we don’t have a crystal ball for hiring, we do have objective, data-driven approaches improve our chances. When you consider the sheer amount of hiring decisions that organizations make (large corporations make thousands or tens of thousands per year), the practical difference between 70/30 and 56/44 odds is absolutely enormous. For companies that commonly have small profit margins, such a significant improvement in their hiring decisions at scale would represent a massive opportunity to create competitive advantage.
Solutions, or how Google hired all the best people already
Google is an example of an organization that has enjoyed a competitive advantage on talent. In Work Rules, former Google CHRO Lazlo Bock described how they took a data-driven approach to hiring. This approach is easily replicable for other organizations and can be summed up as follows:
Spend time defining the problem: Most job descriptions are vague shopping-lists of all of the qualities a hiring manager would like an ideal candidate to have. Spending some time clearly defining (or better yet - using research and analytics to uncover) the most crucial characteristics for performance in the job will lead to the most effective hiring methods
Use structure in interviews to get the best out of human judgment: a full primer on structured interviews is outside the scope of this article, but you should be doing at least these 3 things for your interviews: 1) Ask every candidate the same questions, which are based on the critical competencies you defined. 2) Use anchored rating scales to score candidate responses. 3) Utilize multiple interviewers / raters, but do not discuss scores with each other until after you’ve made them. A combined average of all interviewer ratings creates a data-driven metric to compare candidates, and will be a more accurate predictor than one hiring manager’s judgement.
Leverage ALL data in the final decision: A common mistake is to only look at the most recent interview/assessment to make the final judgment. This is a missed opportunity to combine all of available data and increase your odds of picking the best candidate (shoot for that PC of 70% or more!)