Short Talk

by Carl Bohman

Richard Feynman's Definition of a Good Experiment

Introduction

Since before the dawn of science fair projects, people have been tinkering, testing, experimenting. Whether watching a rat in a maze or electrons in a cyclotron, dropping balls from the Tower of Pisa or probes into Jupiter, an experiment is being performed. Some experiments are legendary (such as the first atomic bomb). Some are very basic (think; science fair volcanoes) and some more definitely not (as those that demonstrate the theories of quantum mechanics).

But not all experiments are created equal. Some are "better" than others. That is some are good, some are bad, and the vast majority are somewhere in between. So what makes a good experiment?

Dr. Richard Feynman, ever the scientist, has many stories about his own experiments. He began in his childhood lab in Rockaway. This expanded in college and exploded into Los Alamos. Soon everywhere he went, the entire universe (so far as he could access it) became his laboratory. With such vast experience in science, surely he knows something of what makes experiments great. According to Dr. Feynman, the key characteristics of a good experiment include at least three main factors: they are well controlled, reproducible, and honest.

Body of Talk

Unless an experiment is well controlled, it is of questionable value. Dr. Feynman was very good at controlling his experiments overall. On more than one occasion, he was ferrying ants around to test their sense of geometry, direction, etc. He described his first ant experiment as follows. [Quote #1] Sometimes, however, he would experience a lapse in thoroughness resulting in wasted time and effort and occasionally costing him the entire experiment. Perhaps the best example Feynman gave of this problem is described this way. [Quote #2]

Experiments should also be reproducible. Dr. Feynman went to great lengths on numerous occasions to retest this or that theory or experiment, proving to himself whether or not they were valid. As one example, he related this story. [Quote #3] He explains the complex reasoning behind the psychologist's claim, then continues. [Quote #4] He was never able to reproduce the results of the psychologist's experiment. Dr. Feynman once tried to explain the importance of replicating an experiment to a student at Comell. [Quote #5] Unfortunately, the student was not permitted to do as Feynman recommended -- which leads directly into the third facet of a good experiment.

Above all else, an experiment must be honest. That is, it must have what Dr. Feynman calls "scientific integrity" (which is actually more than just being honest). Feynman decried honors, awards, recognition, and uniforms. To him they were all (in one way or another) dishonest. Each is there to make one thing appear to be greater or different than it actually is. Thus it is only natural that honesty in science would be so important to him. The student who desired to retest the prior experiment before performing her own was told not to because it "would be wasting time." This cast doubt on the entire experiment and raised questions about the results. Talking about what is often missing in science, particularly what he calls "cargo cult science," Dr. Feynman declares: [Quote #6]. An experiment that does not do that can he missing quite a bit, starting with integrity.

A good experiment does not need to have a profound result or influence. It does not need to be important or even necessarily interesting. A good experiment needs to be well controlled reproducible, and, most of all, honest. Scientists (including amateurs) will almost always learn something in the process of a good experiment. But even if they don't, they should know why.

QUOTE #1 (SYJ, page 93-94)

"In my room at Princeton I had a bay window with a U-shaped windowsill. One day some ants came out on the windowsill and wandered around a little bit. I got curious as to how they found things. I wondered, how do they know where to go? Can they tell each other where food is, like bees can? Do they have any sense of geometry?

This is all amateurish; everybody knows the answer, but I didn't know the answer, so the first thing I did was to stretch some string across the U of the bay window and hang a piece of folded cardboard with sugar on it from the string. The idea of this was to isolate the sugar from the ants, so they wouldn't find it accidentally. I wanted to have everything under control."

QUOTE #2 (SYJ, page 74-75)

"It would have been a fantastic and vital discovery if I had been a good biologist. But I wasn't a good biologist. We had a good idea, a good experiment, the right equipment, but I screwed it up: I gave her infected ribosomes -- the grossest possible error that you could make in an experiment like that. My ribosomes had been in the refrigerator for almost a month, and had become contaminated with some other living things. Had I prepared those ribosomes promptly over again and given them to her in a serious and careful way, with everything under control, that experiment would have worked, and we would have been the first to demonstrate the uniformity of life: the machinery of making proteins, the ribosomes, is the same in every creature. We were there at the right place, we were doing the right things, but I was doing things as an amateur -- stupid and sloppy."

QUOTE #3 (WDYC, page 54-55)

"When I was in graduate school at Princeton a kind of dumb psychology paper came out that stirred up a lot of discussion. The author had decided that the thing controlling the "time sense" in the brain is a chemical reaction involving iron. I thought to myself, "Now how the [heck] could he figure that?"

QUOTE #4 (WDYC, page 55)

"Well it all seemed like a lot of baloney to me -- there were so many things that could go wrong in his long chain of reasoning. But it was an interesting question: what does determine the "time sense"? When you're trying to count at an even rate, what does that rate depend on? And what could you do to yourself to change it?

I decided to investigate. ..."

QUOTE#5 (SYJ, page 344)

"I explained to her that it was necessary first to repeat in her laboratory the experiment of the other person -- to do it under condition X to see if she could also get result A, and then change to Y and see if A changed. Then she would know that the real difference was the thing she thought she had under control."

QUOTE #6 (SYJ, page 341)

"...It's a kind of scientific integrity, a principle of scientific thought that corresponds to a kind of utter honesty -- a kind of leaning over backwards. For example, if you're doing an experiment, you should report everything that you think might make it invalid -- not only what you think is right about it: other causes that that could possibly explain the results; and things you thought of that you've eliminated by some other experiment, and how they worked to make sure the other fellow can tell they have been eliminated.

Details that could throw doubt on your interpretation must be given, if you know them. You must do the best you can -- if you know anything at all wrong, or possibly wrong -- to explain it. If you make a theory, for example, and advertise it, or put it out, then you must put down all the facts that disagree with it, as well as those that agree with it. There also a more subtle problem. When you have put a lot of ideas together to make an elaborate theory, you want to make sure, when explaining what it fits, that those things it fits are not just the things that gave you the idea for the theory; but that the finished theory makes something else come out right, in addition.

Conclusion

In summary, the idea is to try to give all of the information to help others to judge the value of your contribution; not just the information that leads to judgment in one particular direction or another.