Alternatives to t-tests

J. Taylor

2024-01-01

Outline

  • Case studies:

    1. Shuttle O-ring incidents

    2. Cognitive load in math problems

  • Robustness and resistance of two-sample \(t\)-tests

  • Transformations

Case study A: O-rings failing to seal fuel

  • Is there a difference between Cool(<65F) and Warm (>65F)?

  • Temperature on launch day: 29F.

What test to use?

  • Samples are really small

  • No sense in which they could be normally distributed…

Permutation test

  • Decide a test statistic \(T\) comparing Cool(<65F) to Warm (>65F). Could be the two-sample \(t\) test statistic

  • Under \(H_0:\) Cool(<65F) has the same distribution Warm (>65F) suppose we:

    1. Shuffle the response in the two samples by a random ordering \(\sigma\) (called a permutation)

    2. Recompute the statistic yielding \(T(\sigma)\)

Permutation test in practice

Build up a reference distribution under \(H_0\)

Histogram of \(t\)-statistic under \(H_0\)

  • Doesn’t look \(t\)-shaped at all… but test is valid. Why?

  • Under \(H_0\): shuffling the response doesn’t change the distribution!

Compute a \(p\)-value

Rank Sum Test

  • One can use any test statistic for permutation test…

  • Only tests null \(H_0\): distributions are the same.

  • What should we use for \(H_a\)?

Case study B: difference in cognitive load

  • Time to solve math problems under two different presentations

  • Small sample size, not normal?

Rank sum test

  • Rank the outcome

  • \(T\) = Sum the ranks in one of the groups (14 of 28 are Modified)

  • Under \(H_0:\) distributions are identical, \(T\) has distribution the sum of 14 ranks in a random permutation…

Rank sum test

  • Using a builtin test

Rank sum test

Null distribution (ignoring ties)

  • R uses a distributional approximation…

Confidence interval

  • What parameter is this estimating?

Shift alternative

  • \(H_a:\) the difference between groups is a simple shift

  • Other interpretations: estimating median of difference…

Paired data

  • There are plenty of other non-parametric tests out there

Signed rank test for paired data (i.e. schizophrenia)

  • \(H_0:\) distributions are the same in each group..

  • \(\implies\) differences are symmetrically distributed around 0!

Symmetric null

  • \(H_0:\) distribution is symmetric around 0.

Symmetric alternative

  • \(H_a:\) distribution is symmetric around \(\delta\).

Example of signed rank schizophrenia

Compare to t.test