It not only must make a difference, but that difference must be statistically significant because some differences could merely occur by chance.
For example if you flip a coin 20 times, you should get close to one-half heads and one-half tails (10/10) since a head or a tail has probability of occurring.
If you get 11 heads and 9 tails, that may be a numerical difference, but that difference may have simply occurred by chance.
It may not be a statistically significant difference.
Statistics are nothing more than a way of measuring the probability that a significant difference truly exists.
A statistically significant difference is expressed in terms of “degrees of confidence” in which you are certain that this difference could only occur by chance 1 in 100 times for example.
All experiments have as their basis what is called “the null hypothesis”. That hypothesis says that whatever you just did made no statistically significant difference. It is merely a chance occurrence.
When you run that experiment your outcome may wind up “failing to reject the null hypothesis”. In another words, your study did not indicate that the null hypothesis was incorrect. Your results are not statistically significant and could have occurred simply by chance.
Let’s say they are 50 people with back pain, divided into two groups of 25.
None of the 50 can stand for more than an hour without pain.
One (group if 25) is the control group to whom you give a placebo injection.
The other (group of 25) is the experimental group to whom you give a true injection.
The injections are your independent variable.
Ability to stand greater than an hour is your dependent variable (dependent upon which injection they received).
You find that 3 in the control group (with the placebo) are now able to stand longer than an hour.
You find that 8 in the experimental group (with true medication) are now able to stand longer than an hour.
You would then have to determine if this difference you observed between the control and experimental groups was due to chance (remember that the null hypothesis says that it is only due to chance) or is the difference statistically significant so that you (fail to accept the null hypothesis) have a valid finding.