A troubling aspect of most social science research is that most such research is based on samples. Recall from our earlier discussion of sampling that there are good ways to draw samples from a population, and there are some really, really bad ones. The good ones tend to be based on randomization. The problem with random samples is that sometimes (although rarely) they do not reflect the population that they are drawn from. For example, it is improbable that a random sample of 100 students from a particular state university would contain only males. Note my use of the word improbable, and the conspicuous fact that I didn’t choose impossible.
The word impossible cannot be used because improbable things do happen sometimes. People are struck by lightning. People win the lottery. People find buried treasure while digging in a field. Regardless of whether the improbable thing is seen as good or bad, such things sometimes happen. Some of these things are completely unpredictable because we simply do not know enough about the phenomenon to say what the probability is. Scientists dislike such things. Things like the lottery make social scientists a lot more comfortable, simply because there is enough information to compute a precise probability of winning (although most scientists would look at the probability and save their money). It is bad enough that you cannot be sure about something, but it is a small comfort to know just how unsure you are.
When it comes to using samples to conduct research, the researcher can never be 100% sure that a particular “treatment” worked as hypothesized. There is always the nagging possibility that the sample was biased (just by random chance) toward making it seem that the treatment worked when it really didn’t.
Let’s go back to a very early section of this text and recall how a simple experiment works. Let’s say, for example, that a pharmacology researcher has developed a new drug that will lower blood pressure. To find out whether the drug works or not, she will conduct an experiment. The simplest way to do this would be to randomly assign people to an experimental group (those that get the new drug) and a control group (those that get a placebo). After administering the drug and the placebo to the appropriate participants in the study, the pharmacologist records the blood pressure of everyone on both groups.
The data are entered into a spreadsheet, and a mean is calculated for both groups. If the control group has a lower mean blood pressure than the experimental group, then it is likely that the drug does not work. If the mean of the treatment group is lower than the mean of the control group, it is evidence that the drug worked as hypothesized. However, there is a second alternative that sticks in the mind of the researcher like a splinter: What if the participants in the experimental group had a lower blood pressure than the participants in the control group just because of the way the groups were randomly assigned?
Most of the time, randomization will take care of this problem, but not all of the time. There is a small chance that the randomization process created a group with low or high blood pressure—purely by chance. The bottom line is that when you conduct well-designed research using samples, it is possible (but improbable) that you will be wrong in concluding that the treatment worked.
Social scientists cannot abide by qualitative descriptions such as a “small chance.” To be satisfied, an exact probability must be established. That is, the social scientist is much more content with the results of the experiment if it can be said that “there is a 5% chance that I am wrong” or that “there is a 1% chance that I am wrong.” The social science research process requires that the researcher “draw a line in the sand” on this point.
That is, the researcher must specify how much of a chance of being wrong is acceptable when saying that “the treatment worked.” The choice of 5% and 1% above was no accident: These are two commonly accepted thresholds among the scientific community. Before we delve deeper into the idea of hypothesis testing, we must first understand a little more about how we know what the chances of being wrong are.
Last Modified: 09/25/2023