Testing is an important activity in the area of IT. The number of available methodologies, frameworks and models underline the fact that testing makes an impact in today’s IT world. That is, if done correctly. In some cases we don’t meet the very basic pre-requisite. Testing activities should always closely correlate with the real world.
A basic example of this we can find in a zillion of men’s rooms all around the world. Presumably the same is true for ladies rooms. However, due to my gender I don’t have any personal experience with that since the age of three. In men’s rooms it is quite likely that you will find a dispenser of paper towels. The towels in there are cleverly folded. The general idea is that you can take one out in a way that presents your successor with the same excellent starting point. There is always a part of the paper towel hanging out to be grabbed easily. The only thing is: it doesn’t work.
I can see how intelligent designers optimized the dispenser in all kind of ways. And nowadays it is not even necessary anymore to open the thing up to find out how many paper towels are still in there. And in their laboratory or design office, the towels probably come out nicely. But this is not a representative test. The fact that you can take paper towels out of a completely filled dispenser with dry (!) hands has no correlation whatsoever with the real life situation in a men’s room.
So whenever you are building a progression or regression test set-up, make sure your test environment is as close to the real situation as possible. I recently stumbled upon an example with a Disaster Recovery (DR) test. The test would only work with specific components of the production environment included. While the non-availability of this production environment is the whole point of invoking DR in the first place. And that is but one example, there are many more and the paper towel dispenser is there to remind us every day.