experiments

Experiments are setups to test predicitons about the behavior of systems.

In software development, tests are automated experiments to verify that the behavior of the system has not changed in unwanted ways, after having changed the code and the "rules" of the system. Changes are meant to be additive only or to change behavior in predicted ways.

Since writing "tests" is shorter and writing actual code and tests that test them is easier than setting and showing how experiments are wrong, I will use software from now on. But the epistomological argumentative mistakes apply to regular scientific experiments.

The difficulty with tests, is that for them to be meaningful, the setup, data and actions of running them has to be "reasonably" identical.

analysis of the outcome

There are multiple easy ways to do this wrong. Assuming we have a behavior like a natural law or piece of code we want to test and were running a test to verify:

• Our idea od what the rule means can be wrong.
• The actual test and code can be correct but the outcome of the test may return an unexpected fail or pass.
• The test can be written in a way that does not actually verify the outcome of thr rule.

  	rule:
  	x**0=1
  
  	test(x)
  	return type(x) == number
  	

  The test returns a false positive.
• The description of the environment and setup may be so wrong that an accurate reproduction is impossible.
• The experiment may be so expensive to run that an independent verification is not feasible.