The last two days we have been preparing some automated tests which will be used in production during maintenance.
Effectively these tests will verify if all the web and application servers are back up and connected to all the services in the backend systems. During this maintenance servers will be pulled out of the overall production pool and one by one, or in batches, be upgraded to a newer version of something or other, what exactly is not relevant in the context of our tests
Our client being a bank, have all parts of the sites reachable via HTTPS and firewall blocked. In order to reach all the individual webservers directly we have to circumvent the use of the loadbalancers.
In our test automation setup we have made the assumption that all URL’s are served over SSL. Now in preparing these production tests we have found out that this is not always true, for example when bypassing the loadbalancers and firewalls and running the automated tests directly against the individual webservers. So we have had to come up with a new feature in our automation suite: toggling the use of SSL. This is all nice, but it does create a bit of an issue while testing the new suite. Normally when I am creating something to be run on production I try to wait as long as possible before actually running it on production, e.g. debugging my test I do on a test environment rahter than on production. In this case however I have had to debug the tests on production, behind firewalls and SSL layers, since the test environments all require SSL in order to reach them.
Another typical headache I ran into is that some people deem it necessary to make hard links rather than relative links. The objective of one of the tests was to login to a certain account and navigate through the site, like a user would to a different part of the site. The actions would guide us from a new part to an older part of the sites, so in other words, going from asp to aspx. While trying this we found out the new aspx parts have hard links rather than relative links, thus kicking us out from in between firewall and webserver and into the loadbalancer, thus killing our session and breaking the tests.
We were asked to make the tests specifically so that the navigation flow moves from the new code into the old code, since endusers seem to have issues with this transition so now and again and there were hopes that this test could possibly surface these inconsistent issues users see in production, since it has not been possible to reproduce the issue in the test environments.
All of this made me think more about the different strategies one would need to come up with when testing the full chain of a product moving from development environment over to a test environment and then on into production. How valid are the automated tests we have done in test, can we compare them to results coming out of production? Can we run the same tests in production as we do in test? Should the test environment be updated in order to represent production even more? Should the binaries that are deployed in test be 100% identical to the ones in production?
I would like to think that the tests we have done in the test environment are indeed valid and judging by the fact that most of these tests run succesfully on production outside the firewall, e.g. the way our end users reach all functionality, I do believe the tests are indeed valid. It does show clearly to me however, that while testing we do miss some things, such as hard links rather than relative links, which may result in loss of a session for example. Question is how we will be able to catch these links earlier on, rather than in production.
There is still quite some work to be done before we can safely say we have it all covered. Building this fairly simple test suite in FitNesse and WebDriver have shown me some new things to consider while testing applications, such as where is the SSL layer placed, how can i verify if a url is built relative or hard, should we always want to verify this? How do I play around with hostnames and what can this bring me; in this case it brought me the exiting world of hardcoded URL’s to parts of the site and the insight that the binaries used in the test environment are built differently from the ones going into production.
All in all it was just another day in a testers paradise!