Analysis of variance: Difference between revisions
Jump to navigation
Jump to search
imported>Absolutiva No edit summary |
imported>Tsarivan613 m →The F-test: Formatting per WP:LINKFIRST |
||
| Line 1: | Line 1: | ||
{{short description|Collection of statistical models}} | {{short description|Collection of statistical models}} | ||
{{redirect|ANOVA|other uses|Anova (disambiguation)}} | |||
{{Use dmy dates|date=March 2020}} | {{Use dmy dates|date=March 2020}} | ||
| Line 16: | Line 17: | ||
==Example== | ==Example== | ||
[[File: | [[File:ANOVA no fit.png|thumb|No fit: Young vs old, and short-haired vs long-haired]][[File:ANOVA fair fit.svg|thumb|Fair fit: Pet vs Working breed and less athletic vs more athletic]][[File:ANOVA very good fit.jpg|thumb|Very good fit: Weight by breed]]The analysis of variance can be used to describe otherwise complex relations among variables. A dog show provides an example. A dog show is not a random sampling of the breed: it is typically limited to dogs that are adult, pure-bred, and exemplary. A histogram of dog weights from a show is likely to be rather complicated, like the yellow-orange distribution shown in the illustrations. Suppose we wanted to predict the weight of a dog based on a certain set of characteristics of each dog. One way to do that is to ''explain'' the distribution of weights by dividing the dog population into groups based on those characteristics. A successful grouping will split dogs such that (a) each group has a low variance of dog weights (meaning the group is relatively homogeneous) and (b) the mean of each group is distinct (if two groups have the same mean, then it isn't reasonable to conclude that the groups are, in fact, separate in any meaningful way). | ||
In the illustrations to the right, groups are identified as ''X''<sub>1</sub>, ''X''<sub>2</sub>, etc. In the first illustration, the dogs are divided according to the product (interaction) of two binary groupings: young vs old, and short-haired vs long-haired (e.g., group 1 is young, short-haired dogs, group 2 is young, long-haired dogs, etc.). Since the distributions of dog weight within each of the groups (shown in blue) has a relatively large variance, and since the means are very similar across groups, grouping dogs by these characteristics does not produce an effective way to explain the variation in dog weights: knowing which group a dog is in doesn't allow us to predict its weight much better than simply knowing the dog is in a dog show. Thus, this grouping fails to explain the variation in the overall distribution (yellow-orange). | In the illustrations to the right, groups are identified as ''X''<sub>1</sub>, ''X''<sub>2</sub>, etc. In the first illustration, the dogs are divided according to the product (interaction) of two binary groupings: young vs old, and short-haired vs long-haired (e.g., group 1 is young, short-haired dogs, group 2 is young, long-haired dogs, etc.). Since the distributions of dog weight within each of the groups (shown in blue) has a relatively large variance, and since the means are very similar across groups, grouping dogs by these characteristics does not produce an effective way to explain the variation in dog weights: knowing which group a dog is in doesn't allow us to predict its weight much better than simply knowing the dog is in a dog show. Thus, this grouping fails to explain the variation in the overall distribution (yellow-orange). | ||
| Line 102: | Line 103: | ||
Problems which do not satisfy the assumptions of ANOVA can often be transformed to satisfy the assumptions. | Problems which do not satisfy the assumptions of ANOVA can often be transformed to satisfy the assumptions. | ||
The property of unit-treatment additivity is not invariant under a "change of scale", so statisticians often use transformations to achieve unit-treatment additivity. If the response variable is expected to follow a parametric family of probability distributions, then the statistician may specify (in the protocol for the experiment or observational study) that the responses be transformed to stabilize the variance.<ref>Hinkelmann and Kempthorne (2008, Volume 1, Section 6.10: Completely randomized design; Transformations)</ref> Also, a statistician may specify that logarithmic transforms be applied to the responses which are believed to follow a multiplicative model.<ref name="Cox" /><ref>Bailey (2008)</ref> | The property of unit-treatment additivity is not invariant under a "change of scale", so statisticians often use transformations to achieve unit-treatment additivity. If the response variable is expected to follow a parametric family of probability distributions, then the statistician may specify (in the protocol for the experiment or observational study) that the responses be transformed to stabilize the variance.<ref>Hinkelmann and Kempthorne (2008, Volume 1, Section 6.10: Completely randomized design; Transformations)</ref> Also, a statistician may specify that logarithmic transforms be applied to the responses which are believed to follow a multiplicative model.<ref name="Cox" /><ref>Bailey (2008)</ref> | ||
According to Cauchy's | According to [[Cauchy's functional equation]] theorem, the [[logarithm]] is the only continuous transformation that relates multiplication operations to addition operations over the real numbers.<ref>{{cite web | title=Cauchy Functional Equation | url=https://mathworld.wolfram.com/CauchyFunctionalEquation.html }}</ref><ref>{{cite book |last1=Cauchy |first1=Augustin Louis |title=Cours d'analyse de l'École royale polytechnique |trans-title=Analysis course at the Royal Polytechnic School |language=fr |date=1821 |publisher=Imprimerie royale |ol=6477125M |pages=106-109|url=https://archive.org/details/coursdanalysede00caucgoog/}}</ref><ref>{{cite book |title=Lectures on Functional Equations their and Applications |chapter=Solution of Equations by Determining the Values of the Unknown Function on a Dense Set |series=Mathematics in Science and Engineering |date=1966 |volume=19 |pages=31–139 |doi=10.1016/S0076-5392(09)60208-3 |isbn=978-0-12-043750-4 }}</ref> | ||
==Characteristics== | ==Characteristics== | ||
| Line 109: | Line 110: | ||
==Algorithm== | ==Algorithm== | ||
The calculations of ANOVA can be characterized as computing a number of means and variances, dividing two variances and comparing the ratio to a handbook value to determine statistical significance. Calculating a treatment effect is then trivial: "the effect of any treatment is estimated by taking the difference between the mean of the observations which receive the treatment and the general mean".<ref>Cochran & Cox (1992, p 49)</ref> | The calculations of ANOVA can be characterized as computing a number of means and variances, dividing two variances and comparing the ratio to a handbook value to determine statistical significance. Calculating a treatment effect is then trivial: "the effect of any treatment is estimated by taking the difference between the mean of the observations which receive the treatment and the general mean".<ref>Cochran & Cox (1992, p 49)</ref> | ||
===Partitioning of the sum of squares=== | ===Partitioning of the sum of squares=== | ||
| Line 140: | Line 140: | ||
There are two methods of concluding the ANOVA hypothesis test, both of which produce the same result: | There are two methods of concluding the ANOVA hypothesis test, both of which produce the same result: | ||
* The textbook method is to compare the observed value of F with the critical value of F determined from tables. The critical value of F is a function of the degrees of freedom of the numerator and the denominator and the significance level (''α''). If F ≥ F<sub>Critical</sub>, the null hypothesis is rejected. | * The textbook method is to compare the observed value of F with the critical value of F determined from tables. The critical value of F is a function of the degrees of freedom of the numerator and the denominator and the significance level (''α''). If F ≥ F<sub>Critical</sub>, the null hypothesis is rejected. | ||
* The computer method calculates the probability (p-value) of a value of F greater than or equal to the observed value. The null hypothesis is rejected if this probability is less than or equal to the significance level (''α''). | * The computer method calculates the probability ([[p-value|''p''-value]]) of a value of F greater than or equal to the observed value. The null hypothesis is rejected if this probability is less than or equal to the significance level (''α''). | ||
The ANOVA ''F''-test is known to be nearly optimal in the sense of minimizing false negative errors for a fixed rate of false positive errors (i.e. maximizing power for a fixed significance level). For example, to test the hypothesis that various medical treatments have exactly the same effect, the | The ANOVA ''F''-test is known to be nearly optimal in the sense of minimizing false negative errors for a fixed rate of false positive errors (i.e. maximizing power for a fixed significance level). For example, to test the hypothesis that various medical treatments have exactly the same effect, the ''F''-test's ''p''-values closely approximate the [[permutation test]]'s ''p''-values: The approximation is particularly close when the design is balanced.<ref name="HinkelmannKempthorne" /><ref>Hinkelmann and Kempthorne (2008, Volume 1, Section 6.7: Completely randomized design; CRD with unequal numbers of replications)</ref> Such permutation tests characterize [[uniformly most powerful test|tests with maximum power]] against all [[alternative hypothesis|alternative hypotheses]], as observed by [[Paul R. Rosenbaum|Rosenbaum]].<ref group="nb">Rosenbaum (2002, page 40) cites Section 5.7 (Permutation Tests), Theorem 2.3 (actually Theorem 3, page 184) of [[Erich Leo Lehmann|Lehmann]]'s ''Testing Statistical Hypotheses'' (1959).</ref> The ANOVA ''F''-test (of the null-hypothesis that all treatments have exactly the same effect) is recommended as a practical test, because of its robustness against many alternative distributions.<ref>Moore and McCabe (2003, page 763)</ref><ref group="nb">The ''F''-test for the comparison of variances has a mixed reputation. It is not recommended as a hypothesis test to determine whether two ''different'' samples have the same variance. It is recommended for ANOVA where two estimates of the variance of the ''same'' sample are compared. While the ''F''-test is not generally robust against departures from normality, it has been found to be robust in the special case of ANOVA. Citations from Moore & McCabe (2003): "Analysis of variance uses F statistics, but these are not the same as the F statistic for comparing two population standard deviations." (page 554) "The F test and other procedures for inference about variances are so lacking in robustness as to be of little use in practice." (page 556) "[The ANOVA ''F''-test] is relatively insensitive to moderate nonnormality and unequal variances, especially when the sample sizes are similar." (page 763) ANOVA assumes homoscedasticity, but it is robust. The statistical test for homoscedasticity (the ''F''-test) is not robust. Moore & McCabe recommend a rule of thumb.</ref> | ||
is not recommended as a hypothesis test to determine whether two ''different'' samples have the same variance. It is recommended for ANOVA where two estimates of the variance of the ''same'' sample are compared. While the ''F''-test is not generally robust against departures from normality, it has been found to be robust in the special case of ANOVA. Citations from Moore & McCabe (2003): "Analysis of variance uses F statistics, but these are not the same as the F statistic for comparing two population standard deviations." (page 554) "The F test and other procedures for inference about variances are so lacking in robustness as to be of little use in practice." (page 556) "[The ANOVA ''F''-test] is relatively insensitive to moderate nonnormality and unequal variances, especially when the sample sizes are similar." (page 763) ANOVA assumes homoscedasticity, but it is robust. The statistical test for homoscedasticity (the ''F''-test) is not robust. Moore & McCabe recommend a rule of thumb.</ref> | |||
===Extended algorithm=== | ===Extended algorithm=== | ||
| Line 213: | Line 212: | ||
Some popular designs use the following types of ANOVA: | Some popular designs use the following types of ANOVA: | ||
*[[One-way ANOVA]] is used to test for differences among two or more [[statistical independence|independent]] groups (means), e.g. different levels of urea application in a crop, or different levels of antibiotic action on several different bacterial species,<ref>{{Cite web|url=http://www.biomedicalstatistics.info/en/multiplegroups/one-way-anova.html|archive-url=https://web.archive.org/web/20141107211953/http://www.biomedicalstatistics.info/en/multiplegroups/one-way-anova.html|url-status=dead|title=One-way/single factor ANOVA|archive-date=7 November 2014}}</ref> or different levels of effect of some medicine on groups of patients. However, should these groups not be independent, and there is an order in the groups (such as mild, moderate and severe disease), or in the dose of a drug (such as 5 mg/mL, 10 mg/mL, 20 mg/mL) given to the same group of patients, then a [[linear trend estimation]] should be used. Typically, however, the one-way ANOVA is used to test for differences among at least three groups, since the two-group case can be covered by a [[t-test]].<ref>{{Cite journal | doi = 10.1093/biomet/6.1.1 | title = The Probable Error of a Mean | journal = Biometrika | volume = 6 | pages = 1–25 | year = 1908 | *[[One-way ANOVA]] is used to test for differences among two or more [[statistical independence|independent]] groups (means), e.g. different levels of urea application in a crop, or different levels of antibiotic action on several different bacterial species,<ref>{{Cite web|url=http://www.biomedicalstatistics.info/en/multiplegroups/one-way-anova.html|archive-url=https://web.archive.org/web/20141107211953/http://www.biomedicalstatistics.info/en/multiplegroups/one-way-anova.html|url-status=dead|title=One-way/single factor ANOVA|archive-date=7 November 2014}}</ref> or different levels of effect of some medicine on groups of patients. However, should these groups not be independent, and there is an order in the groups (such as mild, moderate and severe disease), or in the dose of a drug (such as 5 mg/mL, 10 mg/mL, 20 mg/mL) given to the same group of patients, then a [[linear trend estimation]] should be used. Typically, however, the one-way ANOVA is used to test for differences among at least three groups, since the two-group case can be covered by a [[t-test]].<ref>{{Cite journal | doi = 10.1093/biomet/6.1.1 | title = The Probable Error of a Mean | journal = Biometrika | volume = 6 | pages = 1–25 | year = 1908 | hdl = 10338.dmlcz/143545 }}</ref> When there are only two means to compare, the [[t-test]] and the ANOVA [[F-test|''F''-test]] are equivalent; the relation between ANOVA and ''t'' is given by {{math|1=''F'' = ''t''<sup>2</sup>}}. | ||
*[[Factorial experiment|Factorial]] ANOVA is used when there is more than one factor. | *[[Factorial experiment|Factorial]] ANOVA is used when there is more than one factor. | ||
*[[Repeated measures]] ANOVA is used when the same subjects are used for each factor (e.g., in a [[longitudinal study]]). | *[[Repeated measures]] ANOVA is used when the same subjects are used for each factor (e.g., in a [[longitudinal study]]). | ||
| Line 273: | Line 272: | ||
*[[Multivariate analysis of covariance]] ('''MANCOVA''') | *[[Multivariate analysis of covariance]] ('''MANCOVA''') | ||
*[[Permutational analysis of variance]] | *[[Permutational analysis of variance]] | ||
*[[Principal component analysis]] | |||
*[[Variance decomposition]] | *[[Variance decomposition]] | ||
{{div col end}} | {{div col end}} | ||
| Line 326: | Line 326: | ||
| publisher = Wiley | location = New York | | publisher = Wiley | location = New York | ||
| year = 2001 | edition = 5th | isbn = 978-0-471-31649-7}} | | year = 2001 | edition = 5th | isbn = 978-0-471-31649-7}} | ||
* [[David S. Moore|Moore, David S.]] & McCabe, George P. (2003). Introduction to the Practice of Statistics (4e). W H Freeman & Co. {{ISBN|0-7167-9657-0}} | * [[David S. Moore|Moore, David S.]] & [[George P. McCabe|McCabe, George P.]] (2003). Introduction to the Practice of Statistics (4e). W H Freeman & Co. {{ISBN|0-7167-9657-0}} | ||
* [[Paul R. Rosenbaum|Rosenbaum, Paul R.]] (2002). ''Observational Studies'' (2nd ed.). New York: Springer-Verlag. {{ISBN|978-0-387-98967-9}} | * [[Paul R. Rosenbaum|Rosenbaum, Paul R.]] (2002). ''Observational Studies'' (2nd ed.). New York: Springer-Verlag. {{ISBN|978-0-387-98967-9}} | ||
* {{cite book |title=The Analysis of Variance | * {{cite book |title=The Analysis of Variance | ||
| Line 333: | Line 333: | ||
|location=New York | |location=New York | ||
|publisher=Wiley |year=1959}} | |publisher=Wiley |year=1959}} | ||
*{{cite book | last = Stigler | first = Stephen M. |author-link=Stephen Stigler| title = The history of statistics : the measurement of uncertainty before 1900 | publisher = Belknap Press of Harvard University Press | location = Cambridge, Mass | year = 1986 | isbn = 978-0-674-40340-6 | url-access = registration | url = https://archive.org/details/historyofstatist00stig }} | *{{cite book | last = Stigler | first = Stephen M. |author-link=Stephen Stigler| title = The history of statistics : the measurement of uncertainty before 1900 | publisher = Belknap Press of Harvard University Press | location = Cambridge, Mass | year = 1986 | isbn = 978-0-674-40340-6 | url-access = registration | url = https://archive.org/details/historyofstatist00stig |oclc=1422544327 }} | ||
* {{ | * {{cite journal |last1=Wilkinson |first1=Leland |author-link=Leland Wilkinson |title=Statistical methods in psychology journals: Guidelines and explanations |journal=American Psychologist |date=August 1999 |volume=54 |issue=8 |pages=594–604 |doi=10.1037/0003-066X.54.8.594 }} | ||
| | |||
|author-link= Leland Wilkinson | |||
|title = Statistical | |||
|journal = American Psychologist | |||
|volume = | |||
|issue = 8 | |||
|pages = 594–604 | |||
|doi = 10.1037/0003-066X.54.8.594 | |||
}} | |||
==Further reading== | ==Further reading== | ||
| Line 390: | Line 380: | ||
|isbn=978-0-387-95361-8 | |isbn=978-0-387-95361-8 | ||
}} | }} | ||
* {{cite book | * {{cite book |title=Block Designs: A Randomization Approach |series=Lecture Notes in Statistics |date=2000 |volume=150 |doi=10.1007/978-1-4612-1192-1 |isbn=978-0-387-98578-7 |first1=Tadeusz |last1=Caliński |first2=Sanpei |last2=Kageyama }} | ||
*{{cite book |last1=Cox |first1=D.R. |last2=Reid |first2=Nancy |title=The Theory of the Design of Experiments |date=2000 |doi=10.1201/9781420035834 |isbn=978-0-429-12628-4 }} | |||
|title=Block | |||
|series=Lecture Notes in Statistics | |||
|volume=150 | |||
| | |||
|isbn=978-0-387-98578-7 | |||
| | |||
| | |||
}} | |||
*{{cite book | |||
|last1=Cox |first1= | |||
|last2=Reid |first2=Nancy | |||
|title= | |||
| | |||
| | |||
|isbn=978- | |||
}} | |||
* {{cite book | * {{cite book | ||
|last1=Hettmansperger |first1=T. P. | |last1=Hettmansperger |first1=T. P. | ||
| Line 477: | Line 448: | ||
|doi=10.1093/biomet/40.3-4.318 | |doi=10.1093/biomet/40.3-4.318 | ||
}} | }} | ||
* {{cite journal | * {{cite journal |last1=Fisher |first1=R. A. |title=Studies in crop variation. I. An examination of the yield of dressed grain from Broadbalk |journal=The Journal of Agricultural Science |date=April 1921 |volume=11 |issue=2 |pages=107–135 |id={{CORE output|162101431}} |doi=10.1017/S0021859600003750 |hdl=2440/15170 |hdl-access=free }} | ||
| | |||
|title=Studies in | |||
|journal=Journal of Agricultural Science | |||
|volume=11 | |||
|issue=2 | |||
|pages=107–135 | |||
|hdl=2440/15170 | |||
|hdl-access=free | |||
}} | |||
==External links== | ==External links== | ||