I'm not an expert, but here's how I would define them: "Experiment", as a noun, simply describes a test under controlled conditions. You could say, "the laboratory performed experiments on guinea pigs." "Experimentation" is the act of experimenting. You could say, "after a long day of experimentation, I finally managed to bake the perfect cake."

experimentation is the act of conducting an experiment (a controlled test); experimentation = running an experiment