As set forth in Rule 23 of the Federal Rules of Civil Procedure, before a class can be certified, plaintiffs must prove, among other things, that common questions of law and fact predominate over individualized questions.
To meet this prerequisite, plaintiffs must set forth classwide, as opposed to individualized, proof of causation and/or damages (both the existence of harm and possibly the amount). Regression analysis is a statistical tool often offered by plaintiffs to establish causation and/or damages using common (as opposed to individualized) proof. 1“Multiple regression involves a variable to be explained—called the dependent variable—and additional explanatory variables that are thought to produce or be associated with changes in the dependent variable.” Daniel L. Rubinfeld, Reference Guide on Multiple Regression, Reference Manual on Scientific Evidence 305 (Federal Judicial Center 3d ed.).
For instance, in oil spill cases, regression analysis is often used by plaintiffs/property owners to demonstrate a correlation 2While regression analyses are often offered as proof of causation, “even the best regression equation cannot … prove causation. The most it can show is a correlation that can give rise to an inference” that causation exists. Morgan v. United Parcel Serv. of Am., Inc., 380 F.3d 459, 466 (8th Cir. 2004); see also Munoz v. Orr, 200 F.3d 291, 301 (5th Cir. 2000) (quoting Tagatz v. Marquette Univ., 861 F.2d 1040, 1044 (7th Cir. 1988) (holding that “statistics can show only correlation and not causation”). between property values and proximity to the oil plume. After controlling for certain major factors that otherwise affect property values (e.g., square footage, age, proximity to parks), plaintiffs’ regression analysis would purport to show that the closer a given property is to the oil plume, the less it is worth, and thus, by inference, the spill caused the diminution in property value.
Regression analysis is also used in antitrust, discrimination, and many other contexts in which plaintiffs need to establish common questions of fact. 3See, e.g., Cordes v. A.G. Edwards & Sons, Inc., 502 F.3d 91, 107 (2d Cir. 2007) (“If the [antitrust] plaintiffs’ [regression analysis] can be employed to make a valid comparison between the but-for fee [the fee if there were no antitrust violation] and the actual fee paid, then it seems to us that the injury-in-fact question is common to the class. Otherwise, it poses individual ones.”). When a class relies upon a regression analysis in support of certification, a fight often ensues over whether the proposed model is sufficiently reliable, and thus admissible, for purposes of proving classwide causation and/or damages.
The Supreme Court endorsed the use of regression analysis in the seminal case of Bazemore v. Friday.
4478 U.S. 385 (1986). Since Bazemore, most courts have largely rubber stamped regression analyses at the class certification stage without looking beyond the model’s surface to determine whether the model was truly sound and reliable. Indeed, since Bazemore, many courts have summarily rejected challenges that defendants made to regression models on the grounds that the models were flawed or did not work, explaining that any flaws go to weight not to admissibility. These courts have rationalized that it is not necessary for the model actually to “work,” so long as it theoretically could be made to work.
This soft scrutiny may be at an end, as the pendulum appears to have swung to a more rigorous approach to class certification. This article traces the evolution of the use of regression analysis in class certification cases, including both the soft and more stringent scrutiny applied throughout the years. It will discuss the implications of Wal-Mart Stores, Inc. v. Dukes,
52011 BL 161238, 79 U.S.L.W. 4527 (U.S. 2011). and Comcast Corporation v. Behrend
62013 BL 80435, 81 U.S.L.W. 4217 (U.S. 2013). on the use of regression analysis, as well as the methods for attacking regression analysis at the class certification stage.
The Bazemore Decision
Initially, in assessing the use of regression analysis for purposes of class certification, the Supreme Court in Bazemore adopted a relatively soft approach. The Fourth Circuit had held that the regression analysis was unacceptable where it did not include all variables thought to have an effect on the matter to be proved by the analysis. The Supreme Court reversed, reasoning:
[W]hile the omission of variables from a regression analysis may render the analysis less probative than it otherwise might be, it can hardly be said, absent some other infirmity, that an analysis which accounts for the major factors must be considered unacceptable as evidenced …. Normally, failure to include variables will affect the analysis’ probativeness, not its admissibility.
7Bazemore, 478 U.S. at 400 (emphasis added).
Notably, the Supreme Court also cautioned that “[t]here may, of course, be some regressions so incomplete as to be inadmissible as irrelevant … .” 8Id. at 400 n.10.
However, this more limiting language was vague and relegated to a footnote. Consequently, as discussed below, it was largely ignored by most courts.
Post-Bazemore Courts Adopt Soft Scrutiny
Approach to Regression Analysis
In the wake of Bazemore, many (if not most) courts almost reflexively accepted regression models proffered by plaintiffs without any rigorous scrutiny of the model’s reliability. These courts summarily rejected defendants’ attacks on the proffered models, citing the broad language quoted above in Bazemore that the model’s flaws “affect [its] probativeness, not its admissibility.” 9To be sure, some courts did conduct a more stringent analysis of regression analyses offered in support of class certification, relying on the footnote in Bazemore that suggested that at some point a model may be so deficient as to be unusable. These cases are discussed below in our overview of methods for attacking regression analysis, as we believe they will likely be given more weight in the future in light of the Supreme Court’s recent Wal-Mart and Comcast rulings.
The court’s analysis in In re Ethylene Propylene Diene Monomer (EPDM) Antitrust Litigation
10256 F.R.D. 82, 100 (D. Conn. 2009). provides a good example of the hands-off approach that many courts have employed post-Bazemore. In re EPDM was an antitrust case in which the plaintiffs proffered a multiple regression analysis to demonstrate that damages could be proven on a classwide basis. Defendants argued that plaintiffs’ analysis failed to account for several factors and variables, highlighting such infirmities with a re-worked version of plaintiffs’ analysis that showed different results. The court rejected the challenge, explaining:
Although characterized as a dispute over the very feasibility of plaintiffs’ analysis, defendants are actually arguing that plaintiffs’ multiple regression analysis, done a slightly different way (i.e., the “right” or “better” way), does not prove what they claim it proves, class wide damages. The defendants claim the model, done slightly differently, demonstrates that at least half the class suffered no damages and thus plaintiffs have not presented a feasible method for proving classwide damages. In essence, the defendants are asking the court to determine which multiple regression model is most accurate, which is ultimately a merits decision … .
The real question before this court is whether the plaintiffs have established a workable multiple regression equation, not whether plaintiffs’ model actually works, because the issue at class certification is not which expert is the most credible, or the most accurate modeler, but rather have the plaintiffs demonstrated that there is a way to prove a class-wide measure of damages through generalized proof. 11Id. at 66-67 (emphasis added).
In re Mercedes-Benz Antitrust Litigation, 213 F.R.D. 180, 190 (D.N.J. 2003), is another example of the hands-off approach to regression models employed by many courts since Bazemore. In that case, the court granted class certification without having analyzed the plaintiffs’ proposed regression model. 12Id. at 189. The court noted only that plaintiffs’ expert claimed that a multiple regression analysis “could establish a reliable, class-wide measure of damages” and the expert’s model had previously been accepted in a different case. 13Id. (emphasis added). The court gave little to no consideration to the model itself or whether the model was appropriate in the case at bar.
Similarly, in In re Bulk [Extruded] Graphite Products Antitrust Litigation, 14No. 02-6030 (D.N.J. April 4, 2006), 2006 BL 47954. the court granted class certification upon finding simply that the plaintiffs had “adduced sufficient evidence and a plausible theory to convince” the court of classwide impact. The court reasoned only that multiple regression is an accepted methodology and that plaintiffs’ expert’s opinion was “bolstered by charts and graphs.” 15Id. In response to the defendants’ criticisms of the plaintiffs’ model, the court refused to “conduct a preliminary inquiry into the merits,” and would not “weigh the arguments of the plaintiffs’ expert and the defendants’ expert.” 16Id. See also In re Dynamic Random Access Memory Antitrust Litig., No. M 02-1486, 2005 BL 152490 (N.D. Cal. June 5, 2006) (granting class certification because plaintiffs had “come forward with seemingly realistic methodologies,” and “[t]he court cannot weigh in on the merits of plaintiffs’ substantive arguments, and must avoid engaging in a battle of expert testimony.”); Hnot v. Willis Group Holdings Ltd.,
228 F.R.D. 476, 484 (S.D.N.Y. 2005) (holding that even if the expert “failed to include all necessary variables in his regression analyses, this affects only the probativeness of the report and not its admissibility.”).
Though not a class certification motion (as the class had already been certified), the court’s analysis (or lack thereof) of plaintiff’s regression model in Cook v. Rockwell
17580 F. Supp. 2d 1071 (D. Colo. 2006). provides another good illustration of how deferential courts have become to regression models that were seemingly flawed on their face. Cook was a class action involving commercial and residential property owners claiming damages resulting from a plutonium spill at a nearby nuclear plant. Plaintiffs relied on a regression analysis to prove a diminution in property value as a result of the spill.
Plaintiffs’ expert developed a computerized model that used multiple regression analysis to identify, quantify, and explain differences in the value of properties allegedly impacted by the spill as compared to properties located away from and unaffected by the spill. The analysis included variables such as physical attributes of the individual properties being analyzed, including the size of the parcel, and the size, age and attributes of any buildings on it; as well as the location-based characteristics of the property, such as neighboring land use, employment levels, demographic data, crime and poverty, traffic volume, accessibility to open space, topography, views of the water, and auto travel time to employment and other locations. 18Cook, 580 F. Supp. 2d. at 1111. Based on the analysis, plaintiffs’ expert claimed to be able to evaluate the loss in property value for each property without resorting to individualized proof.
Defendants argued that plaintiffs’ expert’s analysis was flawed for various reasons, including that it failed to account for an important variable, i.e., whether some or all of the diminution in value reported might be the product of a general concern about the risk of living near a nuclear facility, as opposed to the defendants’ specific operations and the alleged spill that resulted. 19Cook, 580 F. Supp. 2d. at 1112. The court held that, even if it were possible for plaintiffs’ expert to develop such an explanatory variable, its omission does not render the analysis inadmissible. 20Id. at 1113. Citing Bazemore, the court noted that “a regression analysis that includes less than all measurable variables may be sufficient to prove plaintiffs’ case,” and “the weight to be applied to such omitted variables was for the jury to decide.” 21Id.
In sum, the Bazemore decision was interpreted by many—though not all 22Not all courts took this interpretation. See, e.g., Bickerstaff v. Vassar Coll.,
196 F.3d 435, 449-50 (2d Cir. 1999) (“Gray’s regression analysis does not even purport to account for two of these major variables—teaching and services. These variables are too significant not to be accounted for in the regression analysis in this case.”); Sheehan v. Purolator, Inc., 839 F.2d 99, 102-03 (2d Cir. 1988) (affirming denial of class certification where regression analysis was “flawed” because “[t]here were no distinctions for education, prior job history or job level”); Penk v. Oregon State Bd. of Higher Ed., 816 F.2d 458, 465 (9th Cir. 1987) (“Bazemore, however, does not give blanket approval to the introduction of all evidence derived from multiple regression analysis.”).—lower courts as giving them something akin to a license to rubber stamp regression analyses with little to no real scrutiny. As discussed below, however, as a result of the recent pair of decisions by the Supreme Court rejecting regression models, practitioners should anticipate that many lower courts will apply far more rigorous scrutiny to such models going forward.
Supreme Court’s More Rigorous
Approach to Regression Analysis
Walmart
The Supreme Court first signaled a trend of requiring a more stringent review of regression models offered at the class certification stage in Wal-Mart Stores Inc. v. Dukes.
232013 BL 80435, 81 U.S.L.W. 4217 (U.S. 2011). In Wal-Mart, the Supreme Court, without much discussion, rejected the regression models posited by respondents as insufficient to establish the respondents’ legal theory on a classwide basis. The Court did reiterate, however, that “[a] party seeking class certification must affirmatively demonstrate his compliance with [Rule 23].” 24Id. The Court noted that “‘sometimes it may be necessary for the court to probe behind the pleadings before coming to rest on the certification question,’ and that certification is proper only if ‘the trial court is satisfied, after a rigorous analysis, that the prerequisites of Rule 23(a) have been satisfied.’” 25Id. (quoting Gen. Tel. Co. of Southwest v. Falcon, 457 U.S. 147, 160-161 (1982)). Thus, the Court reaffirmed its prior holdings that a “rigorous analysis” must be conducted to determine all of the elements of Rule 23 were satisfied, and explicitly extended this “rigorous analysis” requirement to its review of regression models offered in support of class certification.
Comcast
Apparently, not all courts took heed of the Supreme Court’s holding in Wal-Mart. Just two months after the Supreme Court handed down its decision in Wal-Mart, the U.S. Court of Appeals for the Third Circuit affirmed with little analysis a class certification ruling based upon a regression model. 26See Behrend v. Comcast Corp., 655 F.3d 182 (3d Cir. 2011). The Third Circuit refused to entertain arguments regarding the proposed model simply because those argument happened to also be relevant to the merits of the case. 27Id.
Having apparently had enough of lower courts’ failure to scrutinize regression models offered for class certification, the Supreme Court granted certiorari. The Supreme Court reversed the Third Circuit’s decision and issued—this time—an in-depth opinion on the regression analysis offered, ultimately finding that the model “[could not] possibly establish that damages are susceptible of measurement across the entire class for purposes of Rule 23(b)(3).” 28Comcast.
The Supreme Court in Comcast admonished lower courts to more rigorously analyze the proposed regression model, even if such scrutiny required the lower courts to consider the parties’ arguments that were also pertinent to the merits determination. “For purposes of Rule 23, courts must conduct a ‘rigorous analysis’” to ensure that the regression model relied upon is sufficient. 29Id. The Third Circuit “simply concluded that respondents ‘provided a method to measure and quantify damages on a classwide basis,’ finding it unnecessary to decide ‘whether the methodology [was] a just and reasonable inference or speculative.’ ” 30Id. (internal citations omitted).
The Supreme Court deemed this insufficient. “Under that logic, at the class-certification stage, any method of measurement is acceptable so long as it can be applied classwide, no matter how arbitrary the measurements may be. Such a proposition would reduce Rule 23(b)(3)’s predominance requirement to a nullity.” 31Id.
After Comcast, it is now clear that simply proposing a regression model to measure or quantify damages on a classwide basis is insufficient to achieve class certification. Rather, a regression model offered in support of class certification will need to withstand “rigorous scrutiny,” and the fact that such scrutiny requires the courts to delve into the merits of the action will no longer be a bar to applying a rigorous review.
Since the Comcast decision, lower courts have already begun to undertake a more rigorous analysis of regression models. 32Indeed, since Comcast, the Supreme Court has twice granted certiori, vacated judgment, and remanded for further consideration in light of Comcast, demonstrating that the Court intends to give teeth to the Comcast decision. See Whirlpool Corp. v. Glazer, 2013 BL 85653, 81 U.S.L.W. 3554 (U.S. April 1, 2013) (No. 12-322); Sears, Roebuck & Co. v. Butler, 2013 BL 144566, 81 U.S.L.W. 3669 (U.S. June 3, 2013) (12-1067). For example, in In re High-Tech Employee Antitrust Litigation, 33No. 11-CV-02509, 2013 BL 93558 (N.D. Cal. April 5, 2013). decided less than two weeks after Comcast, the court held “that conducting a thorough review of Plaintiffs’ theory and methodology is consistent with the requirement that the Court conduct a ‘rigorous analysis’ to ensure that the predominance requirement is met.” 34Id. The Court then concluded that plaintiffs failed to satisfy the predominance requirement for proving classwide impact. 35Id. See also Forrand v. Fed. Express Corp., No. CV 08-1360, 2013 BL 147574 (C.D. Cal. April 25, 2013) (denying class certification after conducting “rigorous analysis,” as required under Comcast, and finding that the plaintiff’s methodology failed to adequately tie the plaintiff’s allegations to a reliable measure of damages).
In sum, practitioners should take heed—“rigorous analysis” is the new path forward. Accordingly, practitioners should become familiar with the relatively few decisions in which regression models have been successfully challenged, as those decisions and their rationales will resonate more forcefully in the years to come.
Defeating Class Certification: Methods
for Attacking Regression Analysis
As a result of the Comcast decision, mastery of the tools for attacking regression models is now more important than ever. Below, we discuss some of the more substantial attacks that can—and have been made—on regression analysis.
1. Omitted Variable Bias
Regression analyses may be attacked for failure to account for all of the variables thought to be germane to the issue at hand. Indeed, even in Bazemore, the Supreme Court stated, albeit in a footnote, that there are “some regressions so incomplete as to be inadmissible as irrelevant.” 36Bazemore, 478 U.S. at 400 n.10. See also id. at 400 (noting that regression analysis must at least account for the “major factors”); see also Penk v. Oregon State Bd. of Higher Ed., 816 F.2d 458, 465 (9th Cir. 1987) (“Bazemore, however, does not give blanket approval to the introduction of all evidence derived from multiple regression analysis.”). When important variables are omitted from a model, the model may be biased or its statistical significance may be uncertain.
The key for a defendant is to add the omitted variable, and see if the model regards that variable as statistically significant, and if so, whether the variable changes the results. If it does, omitted variable bias is present. In some cases, the bias may be present to such an extent that the precise variable being tested by the model (e.g., the effect the proximity to an oil spill has on prices) either loses its statistical significance and/or changes directionally (i.e., shows that areas closer to the spill actually have increased the value).
For instance, the Second Circuit in Sheehan v. Purolator
37839 F.2d 99, 103 (2d Cir. 1988). affirmed the district court’s denial of class certification because the regression analysis relied upon was deemed to be “flawed” for failing to take into account non-discriminatory factors, such as education and prior work experience. The court reviewed the factors considered in the regression analysis and the lack of control factors to find that there was a lack of probative evidence supporting the claims. 38Id. See also Freeland v. AT&T Corp., 238 F.R.D. 130, 145 (S.D.N.Y. 2006) (“Where significant variables that are quantifiable are omitted from a regression analysis, … the study may become so incomplete that it is inadmissible as irrelevant.”); Williams v. Boeing Co., No. C98-761P, 2006 BL 7588 (W.D. Wash. Jan. 17, 2006) (finding plaintiffs’ expert’s multiple pools and regression analyses to be unpersuasive because they “do not necessarily compare promotions of similarly-situated employees … [and] tend to group together employees with dissimilar circumstances and locations”); Jones v. GPU. Inc., 234 F.R.D. 82, 94 (E.D. Pa. 2005) (denying certification and stating that “without factoring relevant variables into the [regression] analysis, the statistician leaves open the possibility that those variables, and not racial discrimination, produced differences between African-American and white employees”).
Judge Posner’s opinion in ATA Airlines, Inc. v. Federal Express Corp., 39665 F.3d 882 (7th Cir. 2011). though not a class action, included strong language directed at lower courts everywhere that they cannot simply abdicate their responsibility as gatekeepers when it comes to regression models. At trial, the jury awarded damages based entirely on a regression analysis presented by the plaintiff’s expert witness. 40Id. at 890. Over the defendant’s objection, the trial judge held that the regression analysis was admissible simply because “regression analysis is an accepted model.” 41Id.
On appeal, Judge Posner engaged in a deep-dive analysis of the regression model and easily found numerous inadequacies. In reversing the decision below, he explained that he was not “nitpicking” the model, but rather, that he wished “to remind district judges that, painful as it may be, it is their responsibility to screen expert testimony, however technical … .” 42Id. at 896. Where a model is difficult to comprehend or cannot be readily explained by the parties’ attorneys, Judge Posner suggested that district judges appoint their own expert to assist in understanding the models, or consult readily available written reference materials. 43Judge Posner recommended that district judges consult Daniel L. Rubinfeld, Reference Guide on Multiple Regression, Reference Manual on Scientific Evidence 305 (Federal Judicial Center 3d ed.), or David Cope, Fundamentals of Statistical Analysis (2005). ATA Airlines, 655 F.3d at 890. Judge Posner’s opinion epitomizes stringent analysis of a regression model and what practitioners should come increasingly to expect in the post-Comcast world. 44See also Sheehan v. Daily Racing Form, Inc., 104 F.3d 940, 942 (7th Cir. 1997) (Posner, J.) (holding that a preferred statistical study should be accorded no weight on summary judgment because the statistical analysis demonstrated that expert “fail[ed] to exercise the degree of care that a statistician would use in his scientific work, outside the context of litigation,” and therefore should not have been admitted at trial).
In the wake of Wal-Mart and Comcast, it is safe to say that, although a regression analysis may not be required to include every last variable, courts performing a “rigorous analysis” will be much more receptive to arguments that models should be rejected for omitting numerous and/or significant variables.
2. Explanatory Power of `R-Squared’ Value
Related to the “missing variable” argument is the argument that a regression analysis should be excluded because its low “R-squared” value (also referred to as “R<2>” or “correlation coefficient”) renders it unreliable. “R-squared” is a commonly accepted statistical measure of how well a model explains the data that is being measured by the model. 45See Trout v. Hidalgo, 517 F. Supp. 873, 878 n.6 (D.D.C. 1981) (“The best measure of the degree of explanation of movement of the dependent variable by the model is known as R2.”); Paul J. Hoffman et al., Guide to Regression Analysis, Expert Evidence 361 (1997).
A model’s R-squared value represents the explanatory power of the model. For example, if the alleged injury is a reduction in real property value, an R-squared value of 100 percent means that the model explains all of the observed variations in housing prices. By contrast, an R-squared value of 0 percent means that the model explains none of the variations in housing prices. An R-squared value of 25 percent means that the model explains only 25 percent of the variation in housing prices, and provides no explanation of the remaining 75 percent of housing price variation. 46See Sanner v. Bd. of Trade of City of Chicago, No. 89 C 8467 (N. Dist. Ill. Sept. 28, 2001) (citing Richard A. Wehmhoefer, Statistics in Litigation 83 (McGraw Hill 1985)). Put slightly differently, if one home sold for $100,000 more than another home, an R-squared value of 25 percent means the model can only explain $25,000 of the $100,000 price variation, and cannot account for the remaining $75,000. This is a strong indication that there may be significant variables omitted from the model.
The R-squared value of a given model is a common method for determining whether it is a good fit for the data being analyzed. 47See Matthew Bender 1-15 Scientific Evidence §15.06; see also Daniel L. Rubinfeld, Reference Guide on Multiple Regression, Reference Manual on Scientific Evidence 345 (Federal Judicial Center 3d ed.) (R-squared “provides a measure of the overall goodness of fit of the multiple regression equation”). Ideally, “courts should not rely on regression analyses whose … adjusted R<2> … has been called into serious question.” 48Penk, 816 F.2d at 464 (citations omitted). On the other hand, a high R-squared value could be misleading. 49See 33-SE 4 Moore’s Federal Practice—Civil §V (a large R-squared value “only means that the dependent variable marches in step with the independent one—for any number of possible reasons … .”). “[T]here is always the possibility that [] [a] high R-squared value is a product of happenstance. The R-squared value is evidence of a cause-effect relationship between the independent and dependent variables; but no matter how impressive the number, the R-squared value is not dispositive. The R-squared value may be persuasive circumstantial evidence, but it is not conclusive. The volume of ice cream consumption might be linearly related to the number of homicides because both grow in proportion to the size of the population.” Matthew Bender 1-15 Scientific Evidence §15.06; see also Segar v. Smith, 738 F.2d 1249,1283, n.27 (D.C. Cir. 1984) (“R<2> is not a measure of statistical significance and is not a highly precise indicator of the probative value of a statistical study.”)
Several courts have placed great importance on the R-squared value. 50Other courts, however, have chosen to ignore a low R-squared value, instead relying on what is known as the “t-statistic” value (sometimes referred to as “t-test” or “t-ratio”). Lyman v. St. Jude Med. S.C., Inc., 580 F. Supp. 2d 719, 725 (E. Dist. Wis. 2008) (denying plaintiffs’ Daubert motion and finding that, despite a low R-squared value, “‘the t-statistic’ is a better measure than to determine the reliability of a regression model.”). The t-statistic demonstrates the statistical significance of a model based on its standard error/deviation. David H. Kaye and David A. Freedman, Reference Guide on Statistics (available at: http://www.fjc.gov/public/home.nsf/autoframe?openform&url_l=/public/home.nsf/inavgeneral?openpage&url_r=/public/home.nsf/pages/1448); see also Segar, 738 F.2d at 1261-1262 (finding that the “t-ratio,” not the R-squared value, is the measure of the statistical significance of a regression analysis); Mister v. Illinois Cent. Gulf R.R. Co., 832 F.2d 1427, 1437 (7th Cir. 1987) (relying on the t-test). We discuss statistical significance in the next section below. In Valentino v. United States Postal Service,
51511 F. Supp. 917, 944 (D.D.C. 1981), aff’d, 674 F.2d 56 (D.C. Cir. 1982). for example, during a bench trial the plaintiff introduced a regression model for specified employees that had an R-square value of .284. This R-square meant that 71.6 percent of the variation in salary among these employees was not explained by the regression model. Because many potentially explanatory variables had obviously been omitted from the equation, the Court deemed the regression to have no probative value. 52See also Griffin v. Board of Regents, 795 F.2d 1281, 1292 (7th Cir. 1986) (holding that “the explanatory power of a model is a factor that may legitimately be considered by the district court in deciding whether the model may be relied upon.”).
Accordingly, a too low R-squared value may present an additional method for challenging the validity or reliability of a regression analysis.
3. Lack of Statistical Significance
Regression models should also be rejected where they fail to demonstrate the statistical significance of the data relied on. Statistical significance is the determination of whether changes in data reflect a pattern or simply mere chance. A result is “statistically significant” if the result is one that you would expect to result from chance in only the rarest of circumstances. For example, if you were to flip a coin 1,000 times, you would expect the results to be approximately heads 500 times and tails 500 times. If, instead, your results are heads 800 times and tails 200 times, the results would be statistically significant; they would not be the product of chance. Rather, the results indicate that something else may be wrong with the coin, such as the coin’s relative weight may not be balanced.
Courts will, and should, reject regression models that fail the test of statistical significance. For example, in Boyd v. Interstate Brands Corp.,
53256 F.R.D. 340 (E.D.N.Y. 2009). the court denied plaintiffs’ motion for class certification in an employment discrimination action. To show commonality, plaintiffs had to show that the challenged practice was causally related to a pattern of disparate treatment or had a disparate impact. To demonstrate a causal relation, plaintiffs were required to offer significant statistical proof that the alleged discrimination had an effect on the class as a whole. Although plaintiffs put forth sufficient evidence to show that defendant’s policies were subjective, plaintiffs failed to offer statistical proof demonstrating a causal relation between the challenged policies and a pattern of disparate treatment or impact.
To prove classwide discrimination, “courts require statistically significant proof that the alleged discrimination has had an effect on the class as a whole.” 54Id. Accordingly, plaintiffs’ expert provided a statistical analysis purporting to show disparate treatment. Defendant rebutted plaintiffs’ analysis with its own statistical analysis, including factors that defendant claimed plaintiffs’ expert failed to include. Although the plaintiffs’ analysis yielded a higher standard deviation value than defendants’, the plaintiffs’ analysis was still .02 short of being “statistically significant at conventional test levels.” 55Id.
In other words, the model did not tend to show racial discrimination in employee promotions, demotions, or work assignments. 56Id. Although this result was just below the value needed for statistical significance, it was enough to defeat the model and thus, class certification.
The magistrate ruled that the plaintiffs’ expert failed to “find a statistically significant disparity in promotion rates,” and, as such, the plaintiffs’ statistics could not support a finding that common questions of fact existed among the members of the class. 57Id. The district court adopted the magistrate’s recommendation that class certification be denied, explaining: “plaintiffs’ own expert could not demonstrate statistically significant evidence of discrimination … .” 58Boyd v. Interstate Brands Corp., 256 F.R.D. 340 (E.D.N.Y. March 4, 2009). See also
Piggly Wiggly Clarksville, Inc. v. Interstate Brands Corp., 100 F. App’x 296, 300 (5th Cir. 2004) (affirming denial of class certification where the plaintiffs’ expert failed to persuade the court that “a reliable formula for damages can be devised which will yield statistically significant results.”).
These cases demonstrate that it is important for practitioners attacking regression models to scrutinize those models at the most basic level, including determining whether the results of the model are statistically significant, and relatedly, whether minor changes to the model will cause it to lose its statistical significance. It would not be the first time that an expert omitted variables from his model after realizing that including them would deprive the model of its statistical significance. A practitioner should look for such flaws closely.
4. Multicollinearity
Finally, multicollinearity is an additional, albeit related, basis for attacking a regression model. “Multicollinearity” exists when there are two or more variables that are highly or perfectly correlated (i.e., higher/lower values of one characteristic often occur with higher/lower values of another characteristic). The greater the multicollinearity between two variables, the less likely that a regression analysis will be able to distinguish between competing explanations for movement in the outcome variable. 59Rubinfeld, Reference Guide on Multiple Regression, Reference Manual on Scientific Evidence at 324. Where there is substantial multicollinearity, the size of the sample (large or small) will not matter—the expert will not be able to determine whether there is a relationship between the dependent and independent variables. 60Id. at 325.
For example, where a putative class seeks to demonstrate that their real property values have been diminished by an oil spill, if there are two or more variables that are highly or perfectly correlated, a regression model will not be able to determine which of those variables contributed to the change in property value. Thus, if the oil spill occurred in the same area as another eye sore, such as an industrial park or waste treatment facility, it is likely that the oil spill and the eye sore are highly correlated. In that case, it may not be provable that the property values in the area were reduced as a result of the oil spill as opposed to the other eye sore. Where this is the case, it could be argued that no regression model could be used, as none would work.
Conclusion
Although regression analysis has become a widely accepted methodology for establishing the requirements of Rule 23 at the class certification phase, regression analysis is no longer relatively immune from attack. In light of Comcast and Wal-Mart, lower courts will likely conduct far more searching and in-depth reviews of regression analyses to satisfy the Supreme Court’s mandate that a “rigorous analysis” be performed.
For practitioners offering or attacking regression analyses, it is important to understand the methodology underlying each model, and to carefully present the merits and/or deficiencies of that model to the court. The above tools will prove useful in the coming years as regression analyses undergo this “rigorous analysis.”
