Experiment
(also known as a Recipe or Variation) A specific combination of factor levels (for example, the red headline combined with the small hero shot and the green button) used during a test.
Billy’s explanation:
An experiment is one version of a page in a test. For example, here I have a test matrix with 3 factors and 2 levels each:
Four of the possible combinations or experiments are:
The number of experiments shown in a test and the actual number of experiments possible are not necessarily the same (see fractional factorial test design for more on that.) There are 8 possible experiments in the above recipe matrix, but only 4 shown here.
How do I know there are 8 experiments? Multiply the number of levels of each factor by each other. In this case I have 3 factors with 2 levels each. This makes 2×2x2=8 experiments.
An easy way to remember the definition is to ask, why is it called an experiment? Think about a simple science experiment. Let’s find if baking soda reacts with water or vinegar. In tray 1, we mix vinegar and water, in tray 2 we mix vinegar and baking soda. Tray 1 does nothing, tray 2 bubbles up.
Winner: Tray 2, vinegar
Since we put in different ingredients in each tray, each tray represents an experiment.
In optimization, each page variation is an experiment and we observe it similarly. Each experiment has different ingredients (levels) and the reaction (conversion data) is tracked. It’s a little more complicated than that, but the general idea is the same.
