A few years ago, a close family member was to have a major surgery. In our search for surgeons and hospitals, one important factor we considered was safety records. The Leapfrog Group, a U.S. national nonprofit organization focused on measuring hospitals’ patient safety record, compiles data from nearly 2,000 hospitals. Hospitals are graded on their safety performance based on a composite of metrics, for both process and outcome. Process measurements relate to the efforts hospitals make to enhance safety: handwashing practices, systems for preventing medication mistakes, communication among medical staff, etc. Outcome measures actual safety events: infections, surgical errors, patient falls, etc. As family members of a patient, we realized that outcome was much more important to us than process: We were far more concerned about the risk of infection than about how the hospital gets its staff to wash their hands.
The focus on outcome is what typically drives performance: Athletes do not win medals for their hours of training, but for their performance at races; marketing departments are not credited for how many mailers they send, but on the amount of incremental sales increase resulting from the campaigns; recovery from addiction is not measured by the number of therapy sessions, but by the continuity of sobriety.
In the area of corporate compliance, however, the focus has remained on process, not outcome.
“We trained 95 percent of our employees on compliance.” (But we didn’t know if they learned anything.)
“Our CEO always says ‘do the right thing’!” (But we don’t know if anyone believes him/her, or understands what “do the right thing” even means.)
“We conduct due diligence checks on third parties.” (But we do not monitor their activities to know what they are doing on our behalf.)
The Evaluation of Corporate Compliance Programs, published by the Fraud Section in the Criminal Division of the Department of Justice, has taken the first steps toward asking outcome-based questions: How has the company measured the effectiveness of the training? Has the third-party due diligence ever identified red flags? How has the compliance program been audited and tested? The journey toward true outcome-based measurements, however, still very much continues ahead for corporate compliance professionals. Below are some next steps we might be able take.
Measuring “Tone from the Top”
The outcome of interest here is the tone—culture—of the organization, and the mention of the “top” is based on the belief in the leadership’s role in setting that tone. The prevailing measurements of the number and types of pro-compliance messages uttered by management is somewhat akin to measuring calorie intake by counting calories only when one eats vegetables. At best, such measurements reflect a biased view of tone at the top. An outcome measure, instead, would focus on measuring the culture of the organization, including how leadership commitments are perceived and understood. Credible culture and perception measurements, however, would require much greater rigor than most of the surveys I have seen, which often suffer from vagueness, leading questions, and common self-reporting biases.
Measuring Training Through Behavior, not Entertainment
There has been some shift from measuring attendance rate to measuring how “engaged” the training is. While this is a step in the right direction, “engaged” is often confused with “entertained.” As one practitioner has astutely noted: “[P]eople do not need to do anything differently after consuming entertainment…Compliance training, on the other hand, is a waste of time unless people do something differently afterwards.”
Training is about teaching skills and changing behavior: What needs to be measured, then, are the use of skills and change in behavior.
Measuring Controls As Filters—What “Contaminants” are Getting Past?
I have seen many companies count how many financial or process controls they have, but few count how many control failures there are. Financial and process controls act as filters: Their purpose is to block out the prohibited transactions. When I buy a water filter system, I am less interested in what mechanisms go into the filter system than I am in how many contaminants slip through the system. The outcome measure for the effectiveness of controls, therefore, is the percentage of contaminated transactions that have evaded the controls. Rigorous and continuous audits and monitoring, therefore, are likely the most appropriate measurements.
Ultimately, the outcome of any corporate compliance program is its ability to prevent and detect specific types of misconducts. In that, it is no different from other prevention and detection industries and professions such as public health, safety, and crime prevention. Each of those industries have developed outcome metrics to measure their performance, and may offer valuable lessons for corporate compliance to guide us on our journey towards outcome-based measurements.
Hui Chen (www.HuiChenEthics.com) was the Justice Department’s first-ever compliance counsel expert before leaving in June 2017 to start her own private compliance consulting service. Before she joined the DOJ, Hui served in global senior compliance lead positions at Microsoft, Pfizer, and Standard Chartered Bank.