Cut the “Crud Factor” out of your Employee Listening Data: Psychometric Network Analysis (PNA)
Breaking down the flaws of traditional driver analysis, and recommending a modern approach based on psychometric network analysis (PNA)
Survey data - full of crud?
Of the various corners of people analytics, the annual engagement survey might be the most established – SHRM reports over 80% of organizations conduct one, according to a recent NYT article. Companies ask a huge variety of questions to employees, with surveys commonly containing 80-100, or more questions.
Reporting results to top leadership is straightforward – simply counting the percentage of employees who respond favorably across different survey items can provide a clear readout of what’s going well, and not so well. Unfortunately, making sense of these results and understanding what to do about them is much more difficult and is where most organizations get tripped up.
Driver analysis is the typical approach to identifying specific factors that can move the needle on employee engagement. It attempts to provide more actionable recommendations by parsing out the items that highly-engaged employees respond favorably to, but that differentiate the lower-engaged employees. Confused yet? You’re not alone – driver analysis is difficult to explain, especially to leaders that don’t have a background or spend a lot of time on this stuff.
To make matters worse, the methodology behind driver analysis has some serious flaws. For one, it assumes that the relationships between survey questions are causal (implicit in the “driver” language), despite having no basis to do so – see my last article on quasi-experiments for more on causality pitfalls. But a larger issue with driver analysis is that it relies on correlations, and, well, just about all employee attitudes are correlated with each other. In this tangled web of relationships, the distinction between a survey question that correlates with engagement at .70 vs. another that correlates at .65 may not be a meaningful one.
This gets us to a really interesting stream of psychology literature dealing with this “crud factor” of psychological data, namely the phenomenon of psychological constructs all being correlated. The crud factor creates a messy sort of cosmic background radiation that makes it difficult to tease out true relationships. Any people analytics professional who has looked at their survey data closely can attest to the crud factor’s existence. With the ubiquity of engagement surveys in organizations today, we might need to be thinking about the crud factor as being much more than a niche academic issue. I would argue it’s actually a major contributor to organizations struggling to make sense of, and action on their employee survey data.
Psychometric Network Analysis (PNA)
The statistical methods that are still used to validate survey measures today were created over 100 years ago. This reflects the astonishing success of psychometric techniques and their lasting impact, and by no means are these methods considered to be obsolete. However, in just the past few years a new entrant has shaken things up – psychometric network analysis (PNA). You may be familiar with organizational network analysis (ONA) and the concept here is very similar, except rather than looking at the relationships between people, we’re looking at the relationships between attitudes, personality factors, or other psychometric data.
It’s not always a silver bullet, but one thing that is really exciting about PNA is that it addresses the crud factor. Here’s how: if you visualize a traditional correlation matrix as a network, you would see a very dense network where all survey items are linked (known in graph theory as a “complete” network). This is akin to our chaotic, “cruddy” network that we’re left with when doing driver analysis. Despite the name, it doesn’t actually provide too much insight into what’s driving what, because it’s unable to parse out which items really matter.
Without delving into the technical details, a frequently used method for PNA called adaptive LASSO networks can actually help us prune the network and tease out which of these relationships are essential. Adaptive LASSO networks accomplish this in two ways:
Relationships are conditioned on all other relationships in the network. For example, like in driver analysis we want to know the relationship between X and Y, but because we’re modelling relationships as an interdependent network, we are able to account for the interrelationships between variables A-Z on X, and likewise for Y.
Via the LASSO algorithm, relationships are subject to a penalty, meaning only stronger relationships are preserved, giving us a “sparse” network comprised of fewer, stronger relationships, easing interpretation.
Going from a “cruddy” web of correlations to being able to see unique relationships between X and Y with all else being held constant (ceteris paribus) is extremely helpful. In the employee listening context, it can provide insight into which attitudes are “central” or more important in the network, and which attitudes are closely and uniquely related to “outcome” attitudes like engagement or intent to stay with the organization.
In this example, using fictional data from Keith McNulty’s peopleanalyticsdata package in R, leveraging PNA points us to specific aspects of each survey category that are most closely related to employee happiness. In this example, the first benefits item has the strongest unique relationship with happiness, so we might recommend focusing on that as an action item.
PNA can also lead to unexpected insights, providing a deeper look into relationships between items that we might not have considered based on their surface-level categorization. But most importantly, being able to provide clear and meaningful recommendations to leaders about how to drive specific attitudes higher is something that PNA can do, as opposed to traditional driver analysis.
Other Use Cases / Resources
Survey Development
As mentioned above, PNA offers a substitute for traditional factor analysis methods. That said, method used to statistically validate your survey measures should really depend on the theoretical underpinning. If you’re hypothesizing the existence of some latent factor (i.e., conscientiousness), then you’ll want to stick with factor analysis. However, if you think that the thing we call “conscientiousness” is really just a mix of closely related and interacting attitude elements (a mixture of emotions, beliefs, or other psychological traits), PNA can provide insight into the existence of these relationships.
I am still partial to factor analysis for measures that have a lot of preexisting research behind them, however I would bet that there is going to be a wave of research over the next several years uncovering more complicated and multi-faceted psychological processes that PNA would lend itself better to. See this recent study on creative personalities for an example.
Temperature Analysis
One limitation of traditional survey methods is that they do not consider the amount of attention (salience), or level of importance that individuals assign to different issues. For example, you could have a negative impression of your company’s benefits, but this attitude may not move the needle compared to how you feel about your pay. The good news is that if you’re looking to answer the question “is this issue more or less important to employees now than before?” then temperature analysis is the way to go.
In their primer article published in Nature, a multinational group of psychology researchers introduced the concept of network temperature for longitudinal, or time series data. Network temperature is a measure that elucidates attention through statistical methods, based on the assumption that the interdependence of feelings, beliefs, and behaviors that make up attitudes is greater when an individual directs attention to the attitude(s). Essentially, temperature analysis can look at a psychometric network over time and assign a measure of temperature (attitude) to the network and/or its components.
It is counterintuitive, but a colder network temperature indicates that an attitude or attitudinal group is being subject to more attention, whereas a warmer network temperature suggests less attention. The underlying methodology is actually borrowed from thermodynamics, and you can think of the psychometric measures as being the atoms, and network temperature as measuring the extent to which the atoms constrain each other / are free to roam around.
Additional resources
If you want to learn more about psychometric network analysis or learn how to conduct it yourself, these are some great resources to start: