Open main menu

Changes

General Statistics

2,658 bytes added, 16 October
Updated general statistics wiki page
[[ Category: General ]][[ Category: Data ]][[ Category: Statistics ]]<div class="cell">
<syntaxhighlight lang= General Statistical Methods "r">library(knitr)#figures makde will go to directory called figures, will make them as both png and pdf files opts_chunk$set(fig.path='figures/',dev=c('png','pdf'))options(scipen = 2, digits = 3)# set echo and message to TRUE if you want to display code blocks and code output respectively
There are several important concepts that we will adhere to in our group. These involve design considerationsknitr::knit_hooks$set(inline = function(x) { knitr:::format_sci(x, execution considerations and analysis concerns. The standard for our field is null hypothesis significance testing, which means that we are generally comparing our data to a null hypothesis, generating an 'md''effect size''' and a '''p-value'''. As a general rule, we report both of these both within our Rmd scripts, and in our publications.)})
We generally use an <math>\alpha</math> of <math>p<0.05</math> to determine significance, which means that (if true) we are rejecting the null hypothesis.
superpose.eb <- function (x, y, ebl, ebu =ebl, length = Experimental Design 0.08, ...) arrows(x, y + ebu, x, y - ebl, angle =90, code =3, length = length, ...)
Where possible se <- function(x) sd(x, prior to performing an experiment or study perform a power analysisna. This is mainly to determine the appropriate sample sizes. To do this, you need to know a few of things:rm=T)/sqrt(length(x))
* Either the sample size or the difference. The difference is provided in standard deviations. This means that you need to know the standard deviation of your measurement in question. It is a good idea to keep a log of #load these for your datapackages, so that you can approximate what this is. If you hope to detect a correlation you will need to know the expected correlation coefficient.nearly always needed* The desired false positive rate library(normally 0.05tidyr). This is the rate at which you find a difference where there is none. This is also known as the type I error rate.* The desired power library(normally 0.8dplyr). This indicates that 80% of the time you will detect the effect if there is one. This is also known as 1 minus the false negative rate or 1 minus the Type II error rate.</syntaxhighlight><div class="cell-output cell-output-stderr">
We use the R <pre>Attaching package : 'dplyr''pwr''' to do a power analysis (Champely, 2020). Here is an example:</pre>
</div><div class=== Pairwise Comparasons ==="cell-output cell-output-stderr">
<pre class="r">require(pwr)false.negative.rate &lt;- 0.05statistical.power &lt;- 0.8sd &lt;- 3.5 #this is calculated The following objects are masked from known measurementsdifference &lt;- 3 #you hope to detect a difference pwr.t.test(d = difference, sig.level = false.negative.rate, power=statistical.power)</pre><pre>## ## Two-sample t test power calculation ## ## n = 3.07## d = 3## sig.level = 0.05## power = 0.8## alternative = two.sided## ## NOTE'package: n is number in *each* group</pre>This tells us that in order to see a difference of at least 3, with at standard devation of 3.5 we need at least stats'''3''' observations in each group.:
=== Correlations === filter, lag</pre>
The following is an example for detecting a correlation.</div><div class="cell-output cell-output-stderr">
<pre class="r">require(pwr)false.negative.rate &lt;- 0.05statistical.power &lt;- 0.8correlation.coefficient &lt;- 0.6 #note that this is the r, to get the R2 value you will have to square this result.pwr.r.test(r = correlation.coefficient, sig.level = false.negative.rate, power=statistical.power)</pre><pre>## ## approximate correlation power calculation (arctangh transformation) ## ## n = 18.6## r = 0.6## sig.level = 0.05## power = 0.8## alternative = two.sided</pre>This tells us that in order to detect a correlation coefficient of at least 0.6 (or an R^2 of 0.36) you need more than '''18'The following objects are masked from 'package:base' observations.:
intersect, setdiff, setequal, union</pre>
 
</div>
<syntaxhighlight lang="r"># sets maize and blue color scheme
color.scheme <- c('#00274c', '#ffcb05')</syntaxhighlight>
 
</div>
<span id="general-statistical-methods"></span>
= General Statistical Methods =
 
There are several important concepts that we will adhere to in our group. These involve design considerations, execution considerations and analysis concerns. The standard for our field is null hypothesis significance testing, which means that we are generally comparing our data to a null hypothesis, generating an '''effect size''' and a '''p-value'''. As a general rule, we report both of these both within our Rmd/qmd scripts, and in our publications.
 
We generally use an <math display="inline">\alpha</math> of <math display="inline">p<0.05</math> to determine significance, which means that (if true) we are rejecting the null hypothesis. This is known as null hypothesis significance testing.
 
An alternative approach is to use a Bayesian approach, described in more detail in [https://bridgeslab.github.io/Lab-Documents/Experimental%20Policies/bayesian-analyses.html this document]
 
<span id="pairwise-testing"></span>
= Pairwise Testing =
If you have two groups (and two groups only) that you want to know if they are different, you will normally want to do a pairwise test. This is '''not''' the case if you have paired data (before and after for example). The most common of these is something called a Student's Student’s ''t''-test, but this test has two key assumptions:
* The data are normally distributed
* The two groups have equal variance
<span id="testing-the-assumptions"></span>
== Testing the Assumptions ==
Best practice is to first test for normality, and if that test passes, to then test for equal variance
<span id="testing-normality"></span>
=== Testing Normality ===
To test normality, we use a Shapiro-Wilk test (details on [https://en.wikipedia.org/wiki/Shapiro%E2%80%93Wilk_test Wikipedia] on each of your two groups). Below is an example where there are two groups:
<pre div class="cell"> <syntaxhighlight lang="r">#create seed for reproducibility
set.seed(1265)
test.data &lt;<- tibble(Treatment=c(rep(&quot;"Experiment&quot;",6), rep(&quot;"Control&quot;",6)),
Result = rnorm(n=12, mean=10, sd=3))
#test.data$Treatment &lt;<- as.factor(test.data$Treatment)kable(test.data, caption=&quot;"The test data used in the following examples&quot;")</presyntaxhighlight><div class="cell-output-display"> {|class="wikitable"
|+ The test data used in the following examples
! Treatment
!align="right"| Result
|-
! style="text-align: left;"| ExperimentTreatment|align! style="text-align: right;"| 11.26Result
|-
| style="text-align: left;"| Experiment|alignstyle="text-align: right;"| 811.3326
|-
| style="text-align: left;"| Experiment|alignstyle="text-align: right;"| 98.9433
|-
| style="text-align: left;"| Experiment|alignstyle="text-align: right;"| 119.8394
|-
| style="text-align: left;"| Experiment|alignstyle="text-align: right;"| 611.5683
|-
| style="text-align: left;"| Experiment|alignstyle="text-align: right;"| 116.4156
|-
| Controlstyle="text-align: left;"| Experiment|alignstyle="text-align: right;"| 811.8941
|-
| style="text-align: left;"| Control|alignstyle="text-align: right;"| 118.5989
|-
| style="text-align: left;"| Control|alignstyle="text-align: right;"| 911.3959
|-
| style="text-align: left;"| Control|alignstyle="text-align: right;"| 89.7439
|-
| style="text-align: left;"| Control|alignstyle="text-align: right;"| 68.3174
|-
| style="text-align: left;"| Control|style="text-align: right;"| 6.31|-| style="text-align: left;"| Control| style="text-align: right;"| 7.82
|}
 
</div>
 
</div>
Each of the two groups, in this case '''Test''' and '''Control''' must have Shapiro-Wilk tests done separately. Some sample code for this is below (requires dplyr to be loaded):
<pre div class="cell"> <syntaxhighlight lang="r">#filter only for the control datacontrol.data &lt;<- filter(test.data, Treatment==&quot;"Control&quot;")
#The broom package makes the results of the test appear in a table, with the tidy command
library(broom)
#run the Shapiro-Wilk test on the values
shapiro.test(control.data$Result) %&gt;>% tidy %&gt;>% kable(caption="Shapiro-Wilk test for normality of control data")</presyntaxhighlight>{|!align<div class="rightcell-output-display"| statistic>!align{| class="rightwikitable"| p.value! method|+ Shapiro-Wilk test for normality of control data
|-
! style="text-align: right;"|statistic! style="text-align: right;"| p.value! style="text-align: left;"| method|-| style="text-align: right;"| 0.968|alignstyle="text-align: right;"| 0.88| style="text-align: left;"| Shapiro-Wilk normality test
|}
 <pre class/div><syntaxhighlight lang="r">experiment.data &lt;<- filter(test.data, Treatment==&quot;"Experiment&quot;")shapiro.test(test.data$Result) %&gt;>% tidy %&gt;>% kable(caption="Shapiro-Wilk test for normality of the test data")</presyntaxhighlight>{|!align<div class="rightcell-output-display"| statistic>!align{| class="rightwikitable"| p.value! method|+ Shapiro-Wilk test for normality of the test data
|-
! style="text-align: right;"|statistic! style="text-align: right;"| p.value! style="text-align: left;"| method|-| style="text-align: right;"| 0.93|alignstyle="text-align: right;"| 0.377| style="text-align: left;"| Shapiro-Wilk normality test
|}
 
</div>
 
</div>
Based on these results, since both p-values are &gt;0.05 we do not reject the presumption of normality and can go on. If one or more of the p-values were less than 0.05 we would then use a Mann-Whitney test (also known as a Wilcoxon rank sum test) will be done, see below for more details.
<span id="testing-for-equal-variance"></span>
=== Testing for Equal Variance ===
We generally use the [https://cran.r-project.org/web/packages/car/index.html car] package which contains code for [https://en.wikipedia.org/wiki/Levene%27s_test Levene's Levene’s Test] to see if two groups can be assumed to have equal variance. For more details see @car:
<pre div class="rcell">#load the car packagelibrary(car)
<syntaxhighlight lang="r">#load the car packagelibrary(car)</syntaxhighlight><div class="cell-output cell-output-stderr"> <pre>Loading required package: carData</pre> </div><div class="cell-output cell-output-stderr"> <pre>Attaching package: 'car'</pre> </div><div class="cell-output cell-output-stderr"> <pre>The following object is masked from 'package:dplyr':  recode</pre> </div><syntaxhighlight lang="r">#runs the test, grouping by the Treatment variableleveneTest(Result ~ Treatment, data=test.data) %&gt;>% tidy %&gt;>% kable(caption="Levene's test on test data")</presyntaxhighlight>{|!align<div class="rightcell-output cell-output-stderr"| statistic>!align<pre>Warning in leveneTest.default(y ="right"| py, group = group, .value..): group coerced to!alignfactor.</pre> </div><div class="rightcell-output-display"| df>!align{| class="rightwikitable"| df.residual+ Levene’s test on test data
|-
! style="text-align: right;"|statistic! style="text-align: right;"| p.value! style="text-align: right;"| df! style="text-align: right;"| df.residual|-| style="text-align: right;"| 0.368|alignstyle="text-align: right;"| 0.558|alignstyle="text-align: right;"| 1|alignstyle="text-align: right;"| 10
|}
 
</div>
 
</div>
<span id="performing-the-appropriate-pairwise-test"></span>
== Performing the Appropriate Pairwise Test ==
The logic to follow is:
* If the Shapiro-Wilk test passes, do Levene's Levene’s test. If it fails for either group, move on to a '''Wilcoxon Rank Sum Test'''.* If Levene's Levene’s test ''passes'', do a Student's Student’s ''t'' Test, which assumes equal variance.* If Levene's Levene’s test ''fails'', do a Welch's Welch’s ''t'' Test, which does not assume equal variance.
<span id="students-t-test"></span>== Student's = Student’s ''t'' Test ===
<pre div class="cell"> <syntaxhighlight lang="r">#The default for t.test in R is Welch's, so you need to set the var.equal variable to be TRUEt.test(Result~Treatment,data=test.data, var.equal=T) %&gt;>% tidy %&gt;>% kable(caption="Student's t test for test data")</presyntaxhighlight>{|!align<div class="rightcell-output-display" width="8%"| estimate>!align="right" width="9%"| estimate1!align="right" width="9%"{| estimate2!alignclass="rightwikitable" width="9%"| statistic!align="right" width="7%"| p.value!align="right" width="9%"| parameter!align="right" width="8%"| conf.low!align="right" width="9%"| conf.high!width="16%"| method!width="11%"| alternative+ Student’s t test for test data
|-
! style="text-align: right;"|estimate! style="text-align: right;"| estimate1! style="text-align: right;"| estimate2! style="text-align: right;"| statistic! style="text-align: right;"| p.value! style="text-align: right;"| parameter! style="text-align: right;"| conf.low! style="text-align: right;"| conf.high! style="text-align: left;"| method! style="text-align: left;"| alternative|-| style="text-align: right;"| -1.1|alignstyle="text-align: right;"| 8.79|alignstyle="text-align: right;"| 9.89|alignstyle="text-align: right;"| -0.992|alignstyle="text-align: right;"| 0.345|alignstyle="text-align: right;"| 10|alignstyle="text-align: right;"| -3.56|alignstyle="text-align: right;"| 1.37| style="text-align: left;"| Two Sample t-test| style="text-align: left;"| two.sided
|}
=== Welch's ''t'' Test ===
<pre /div> </div><span id="welchs-t-test"></span>=== Welch’s ''t'' Test === <div class="cell"> <syntaxhighlight lang="r">#The default for t.test in R is Welch's, so you need to set the var.equal variable to be FALSE, or leave the defaultt.test(Result~Treatment,data=test.data, var.equal=F) %&gt;>% tidy %&gt;>% kable(caption="Welch's t test for test data")</presyntaxhighlight>{|!align<div class="rightcell-output-display" width="8%"| estimate>!align="right" width="9%"| estimate1!align="right" width="9%"{| estimate2!alignclass="rightwikitable" width="9%"| statistic!align="right" width="7%"| p.value!align="right" width="9%"| parameter!align="right" width="8%"| conf.low!align="right" width="9%"| conf.high!width="20%"| method!width="10%"| alternative+ Welch’s t test for test data
|-
! style="text-align: right;"|estimate! style="text-align: right;"| estimate1! style="text-align: right;"| estimate2! style="text-align: right;"| statistic! style="text-align: right;"| p.value! style="text-align: right;"| parameter! style="text-align: right;"| conf.low! style="text-align: right;"| conf.high! style="text-align: left;"| method! style="text-align: left;"| alternative|-| style="text-align: right;"| -1.1|alignstyle="text-align: right;"| 8.79|alignstyle="text-align: right;"| 9.89|alignstyle="text-align: right;"| -0.992|alignstyle="text-align: right;"| 0.345|alignstyle="text-align: right;"| 9.72|alignstyle="text-align: right;"| -3.57|alignstyle="text-align: right;"| 1.38| style="text-align: left;"| Welch Two Sample t-test| style="text-align: left;"| two.sided
|}
 
</div>
 
</div>
<span id="wilcoxon-rank-sum-test"></span>
=== Wilcoxon Rank Sum Test ===
<pre div class="cell"> <syntaxhighlight lang="r"># no need to specify anything about variancewilcox.test(Result~Treatment,data=test.data) %&gt;>% tidy %&gt;>% kable(caption="Mann-Whitney test for test data")</presyntaxhighlight>{|!align<div class="rightcell-output-display"| statistic>!align{| class="rightwikitable"| p.value! method! alternative|+ Mann-Whitney test for test data
|-
! style="text-align: right;"|statistic! style="text-align: right;"| p.value! style="text-align: left;"| method! style="text-align: left;"| alternative|-| style="text-align: right;"| 12|alignstyle="text-align: right;"| 0.394| style="text-align: left;"| Wilcoxon rank sum exact test| style="text-align: left;"| two.sided
|}
 
</div>
 
</div>
<span id="corrections-for-multiple-observations"></span>
= Corrections for Multiple Observations =
The best illustration I have seen for the need for multiple observation corrections is this cartoon from XKCD (see http://xkcd.com/882/):
[[File:http://imgs.xkcd.com/comics/significant.png|frame|none|alt<div class=Significance by XKCD. Image is from http://imgs.xkcd.com/comics/significant.png|caption Significance by XKCD. Image is from http://imgs.xkcd.com/comics/significant.png]]"figure">
Any conceptually coherent set of observations must therefore be corrected for multiple observations[[File:http://imgs. In most cases, we will use the method of Benjamini and Hochberg since our p-values are not entirely independentxkcd. Some sample code for this com/comics/significant.png|Significance by XKCD. Image is herefrom http://imgs.xkcd.com/comics/significant.png]]
<pre /div>Any conceptually coherent set of observations must therefore be corrected for multiple observations. In most cases, we will use the method of @Benjamini1995 since our p-values are not entirely independent. Some sample code for this is here: <div class="cell"> <syntaxhighlight lang="r">p.values &lt;<- c(0.023, 0.043, 0.056, 0.421, 0.012)data.frame(unadjusted = p.values, adjusted=p.adjust(p.values, method=&quot;"BH&quot;")) %>% kable(caption="Effects of adjusting p-values by the method of Benjamini-Hochberg")</presyntaxhighlight><prediv class="cell-output-display">##  {| class="wikitable"|+ Effects of adjusting p-values by the method of Benjamini-Hochberg|-! style="text-align: right;"| unadjusted ! style="text-align: right;"| adjusted## 1 |-| style="text-align: right;"| 0.023 | style="text-align: right;"| 0.0575057## 2 |-| style="text-align: right;"| 0.043 | style="text-align: right;"| 0.0700070## 3 |-| style="text-align: right;"| 0.056 | style="text-align: right;"| 0.0700070## 4 |-| style="text-align: right;"| 0.421 | style="text-align: right;"| 0.4210421## 5 |-| style="text-align: right;"| 0.012 | style="text-align: right;"| 0.0575057|}  </div> </div><span id="session-information"></prespan>
= Session Information =
<pre div class="cell"> <syntaxhighlight lang="r">sessionInfo()</presyntaxhighlight><div class="cell-output cell-output-stdout"> <pre>## R version 4.4.1 (2024-06-14)## Platform: x86_64-apple-darwin20## Running under: macOS Monterey 12Sonoma 14.7.6## ## Matrix products: default## BLAS: /Library/Frameworks/R.framework/Versions/4.4-x86_64/Resources/lib/libRblas.0.dylib ## LAPACK: /Library/Frameworks/R.framework/Versions/4.4-x86_64/Resources/lib/libRlapack.dylib; LAPACK version 3.12.0## ## locale:## [1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8## ## time zone: America/Detroit## tzcode source: internal## ## attached base packages:## [1] stats graphics grDevices utils datasets methods base ## ## other attached packages:## [1] car_3.1-2 carData_3.0-5 broom_1.0.6 ## [4] pwr_1.3-0 knitcitations_1.0.12 dplyr_1.1.4 ## [7] tidyr_1.3.1 [6] knitr_1.48 ## ## loaded via a namespace (and not attached):## [1] jsonlite_1vctrs_0.86.8 compiler_45 cli_3.46.1 tidyselect_1.23 rlang_1.1 Rcpp_1.04 xfun_0.13 47 ## [5] xml2_1purrr_1.30.6 stringr_1.52 generics_0.1 jquerylib_0.13 jsonlite_1.4 yaml_28.38 glue_1.10 7.0 ## [9] fastmap_1backports_1.25.0 R6_2 htmltools_0.5.1 plyr_1.8.9 generics_0.1fansi_1.3 0.6 rmarkdown_2.28 ## [13] backports_1abind_1.54-8 evaluate_0.24.0 tibble_3.2.1 RefManageR_1fastmap_1.42.0 lubridate_1.9.3 ## [17] bslib_0yaml_2.83.0 10 pillar_1.9lifecycle_1.0 rlang_1.14 compiler_4.4 utf8_1.21 htmlwidgets_1.6.4 ## [21] stringi_1pkgconfig_2.80.4 cachem_13 rstudioapi_0.116.0 xfun_0digest_0.46 sass_06.437 R6_2.9 5.1 ## [25] bibtex_0tidyselect_1.52.1 timechange_0.3.0 cli_3utf8_1.62.3 withr_34 pillar_1.09.0 ## [29] magrittr_2.0.3 digest_0.6.36 lifecycle_1.0.4 vctrs_0.6.5 ## [3329] evaluate_0.24withr_3.0 glue_1.7.0 abind_1.4-5 1 fansi_1.0.6 ## [37] rmarkdown_2.27 purrr_1.0.2 httr_1.4.7 tools_4.4.1 </pre> ## [41] pkgconfig_2.0.3 htmltools_0.5.8.1</prediv> </div><span id="references"></span>
= References =
<a namediv id=bib-pwr"refs">   </adiv>[[#cite-pwr|[1]]] S. Champely. ''pwr: Basic Functions for Power Analysis''. R package version 1.3-0. 2020. URL: https://CRAN.R-project.org/package=pwr.