A Closer look at testing and intervals

16.6. A Closer look at testing and intervals

  • Sample is clearly not random

  • Complete population

  • Large n - Approximations and unimportant differences

  • Large p - number of tests, contradictions tests

The sampling distribution of a statistic often is close to normal. There are distributions that look like the normal but with thicker tails that are used as well.

A sample of convenience is where the data is not collected according to a sampling design. In some cases, it may be reasonable to proceed as if the data were collected using a random mechanism

The sample is the complete population in which case one might argue that inference is not required since the population and sample values are one and the same. In these situations, we can put a different meaning to the hypothesis tests we are making. We might suppose that if the number of species had no relation to the five geographic variables, then the observed response values would be randomly distributed between the islands without relation to the predictors. We might then ask what the chance would be under this assumption that an F-statistic would be observed as large or larger than one we actually observed.

Thus it is possible to give some meaning to the p-value when the sample is the population or for samples of convenience although one has to be clear that one’s conclusion apply only the particular sample. Tests involving just one predictor also fall within the permutation test framework. We permute that predictor rather than the response.

Another approach that gives meaning to the p-value when the sample is the population involves the imaginative concept of “alternative worlds” where the sample/population at hand is supposed to have been randomly selected from parallel universes. This argument is definitely more tenuous.

A model is usually only an approximation of underlying reality which makes the meaning of the parameters debatable at the very least. We will say more on the interpretation of parameter estimates later but the precision of the statement that beta* = 0 exactly is at odds with the acknowledged approximate nature of the model. Furthermore, it is highly unlikely that a predictor that one has taken the trouble to measure and analyze has exactly zero effect on the response. It may be small but it won’t be zero.

The inference depends on the correctness of the model we use. We can partially check the assumptions about the model but there will always be some element of doubt. Sometimes the data may suggest more than one possible model which may lead to contradictory results.

Statistical significance is not equivalent to practical significance. The larger the sample, the smaller your p-values will be so don’t confuse p-values with a big predictor effect. With large datasets it will be very easy to get statistically significant results, but the actual effects may be unimportant