The topic of organizational culture has been one of enduring and increasing interest in ethics and compliance circles. When widespread misconduct occurs—as in cases such as Wells Fargo’s account fraud or VW’s emission scandals—culture is raised as a key factor in incubating and enabling the compliance failures. Compliance functions often rely on training and internal communications as part of an effort to ensure that the “culture of compliance” of which enforcers and regulators often speak exists and is paramount in employee decision-making. The questions are: How can culture, and changes to it be measured? And how do you know your training and communications are affecting employee comprehension of cultural expectations?

The Problem. Current efforts to measure corporate culture and noncompliance rely on traditional surveys, where employees are asked straightforward questions about their opinions regarding the corporation and its operations, the effectiveness of trainings, and whether they think anything needs to be changed. Although this is a great start, traditional surveys are limited in two main ways: 1) they produce socially desirable responses and 2) their inability to truly establish causal relationships.

1) Social desirability—Whenever you ask someone their opinion, there is a risk that they will tell you what they think you want to know as opposed to providing an honest answer. This is especially a concern when the person responding believes that the person implementing the survey (for example, the human resources department or compliance department) might be able to identify the respondent. In such a situation, the respondent is unlikely to answer honestly for fear that they will face repercussions within their organization.

2) Inability to establish true causality—Let’s say you want to know whether a new communication (e.g., a company-wide memorandum) emphasizing internal disciplinary measures will produce fewer policy violations among your employees. You could release the memo and measure violations afterwards. However, can you be sure that any reduction in violations is due to the emphasis on discipline? Or is it possible that something else is impacting the occurrence of violations? Perhaps, for example, the communication coincided with the winding down of a particularly risky operation and the decrease in risk is what actually decreased violations, not the memo itself? What if an increase in violations results because employees felt devalued or untrusted due to the emphasis on disciplinary measures?

The Solution. In both cases, traditional survey methods are unable to tell you accurately what is having an effect. A better method is what we call “factorial surveys.” Factorial surveys present a hypothetical situation to respondents, and randomly vary certain parts of the scenario to see how those changes impact the outcome of interest.

The hypothetical nature of the scenario helps prevent social desirability—people are more likely to be honest when you’re not asking them to report their own opinions or likelihood of noncompliance. Much research shows that asking about a situation in the third person (e.g., asking about your friends or coworkers) is a great proxy for the likelihood of one’s own offending because we tend to assume that other people are somewhat similar to ourselves.

The randomized component of factorial surveys helps rule out alternative explanations for the relationship of interest. Basically, if people are randomly assigned to see A as opposed to B, we can safely assume that any differences in compliance likelihood between the A group and the B group are due to those conditions and not anything else.

The Example. An “A group” sees a scenario describing immediate job dismissal as a result of suspected noncompliance, while the “B group” sees a scenario describing a disciplinary hearing in which they are given an opportunity to defend themselves before disciplinary actions are taken. Randomly assigning employees to see one of those two scenarios ensures that other factors are not conflated with the outcome of interest.

If you decided not to randomly assign those scenarios to people—opting instead to have the first 50 people take the “A” survey and the last 50 people take the “B” survey—the people who took the survey earlier in the day might be more likely to comply with all rules and requests regardless of potential consequences. As such, they would respond to the survey with an increased likelihood of complying regardless of which group they were in. By failing to randomly assign people to conditions, you would incorrectly conclude that A (immediate job dismissal) produces more compliance than B (disciplinary hearing).

Overall, factorial surveys offer a unique way to determine how likely compliance is among your employees and assess how potential changes to corporate trainings, messaging, incentive structures, etc. might produce more compliance. These surveys also offer a unique way to assess existing training/orientation efforts—instead of asking people to simply recall information they learn during training, you can use scenarios to see whether employees are able to navigate ethically ambiguous situations effectively. Furthermore, these surveys can be designed to address many different areas of concern, measure compliance at all levels of the corporation, and ensure the confidentiality of respondents so they feel safe answering honestly.

Melissa Rorie, Ph.D., is an Assistant Professor in the University of Nevada-Las Vegas Greenspun College of Urban Affairs. She has been researching corporate compliance issues for over a decade, and has more broadly been conducting research in a wide variety of settings (e.g., government agencies, schools, police departments, professional associations) for more than 15 years.

Hui Chen is an independent ethics and compliance consultant and was the Justice Department’s first-ever compliance counsel expert. She had served in global senior compliance lead positions at Microsoft, Pfizer, and Standard Chartered Bank.