We have said that statistics is about using clear thinking, logic, and data we collect to answer questions about the world. There are really two kinds of questions about the world:
If we merely want to know what is happening out in the world, we can simply look. There can still be sophisticated statistical methodology here, for example if it is impractical to get all of the information we want, so we want to draw a conclusion about the entire population from a sample. We will discuss these things later in this course.
If we want to draw conclusions about causality, observations are insufficient. This is because simply seeing B always follow A out in the world does not tell us that A causes B. For example, maybe they are both caused by Z, which we didn't notice had always happened before those A and B, and A is simply a bit faster than B, so it seems always to proceed, even to cause, B. If, on the other hand, we go out in the world and do A and then always see B, we would have more convincing evidence that A causes B.
Therefore, we distinguish two types of statistical studies:
Design of Experiments, which deals, obviously, with experiments and how to do them well, is the topic of this unit.
Measurements always have small, random errors. In addition, we will also intentionally introduce randomness in just a moment. So some mathematical methods in statistics are designed to draw the best possible conclusions despite noisy data. But mathematical methods have more difficulty fixing systematically incorrect data. Therefore what is really the biggest problem for statistical analysis is
Oddly, since we just said that random noise in our data is something that makes us do extra work to clean up the data, randomization is a great tool in experimental design. That's because bias is the real danger so if use randomness wherever possible in our designs — true randomness — we will (pretty much by definition of the word "random") not produce systematically flawed data. In short, randomization is the best tool we have to avoid bias. If some step of an experiment involves a choice and you think you have made it in a perfectly fair, unbiased way, you might simply not have thought of some subtle, unseen connection which causes bias. Better simply to use a random choice at that point of the design, which can never create bias.
If we are doing an experiment to try to understand something in the world, we should not simply do the interesting new treatment to all of our subjects and see what happens. In a certain sense, if we did that, we would simply be changing the whole world (at least the world of all of our test subjects) and then doing an observational study. To really do an experiment, we must compare two treatments.
Therefore any real experiment involves at least two groups:
As we discussed above, when we have to put test subjects into one of these two groups, it is very important to use a selection method which has no bias. The only way to be sure of this is to use a random assignment of subjects to the experimental or control group.
Humans are particularly hard to study, because their awareness of their environments can have surprising effects on what they do and even what happens, physically, to their bodies. This is not because people "fake" the results: there can be real changes in patients' bodies even when you give them a medicine which is not physiologically effective, and real changes in their performance on tests or in athletic events when you merely convince them that they will do better. This beneficial consequence of a treatment which should not directly cause an improvement is called The Placebo Effect.
In the early days of research into the Placebo Effect, the pill that doctors would give to play the role of the "placebo" would look like other pills, but would be made just of sugar (glucose), which (in those quite small quantities) has essentially no physiological consequences and so is a sort of neutral dummy pill. We still call medical placebos "sugar pills" even though now they are often made of some even more neutral material, like the starch binder which is used as a matrix containing the active ingredient in regular pills &mdahs; but without any active ingredient.
Since the Placebo Effect is a real phenomenon with actual, measurable consequences, when making an experimental design and choosing the new treatment and the treatment for the control group, it is important to give the control group something. If they get nothing, they do not have the beneficial effect of the Placebo Effect, so they will not have as good measurements as the experimental group, even if the experimental treatment had no actual useful effect. So we have to equalize for both groups the benefit provided by the Placebo Effect, and give them both an treatment which looks about the same (compare pills to pills, injections to injections to injections, operations to operations, three-hour study sessions in one format to three-hour sessions in another format, etc.) to the subjects.
Here is a very nice short video talking about the placebo effect, starring the excellent researcher, author, and doctor Ben Goldacre. If you dare, here is another of Goldacre's videos which talks about a strange sort of backwards placebo effect called the nocebo effect; be warned that he talks incredibly quickly in this video and also uses quite adult language — avoid this video if four-letter words offend you. [Search YouTube for Ben Goldacre if you have a spare moment. His videos are fascinating, entertaining, and educational about medical research and even statistics.]
It also could be a problem if the experimenter knew who was getting which treatment. Perhaps if the experimenter knew a subject was only getting the control, they would be more compassionate or, alternatively, more dismissive. In either case, the systematically different atmosphere for that group of subjects would again be a possible cause of bias.
Of course, when we say that the experimenter doesn't know which treatment a particular patient is getting, we mean that they do not know that at the time of the treatment. Records must be kept somewhere, and at the end of the experiment, the data is divided between control and experimental groups to see which was effective.
When one party is kept ignorant of the treatment being administered, we say that the information has been blinded. As we have just seen, the best situation is when the experiment is run double-blind, meaning that the neither subjects nor experimenters know who gets which treatment until the end of the experiment (when both must be told, one out of fairness, and one to learn something from the data that was collected).