Selecting performance test tooling – Part 2

In my first installment I wrote about how I got the requirements together. Based on those I wrote a plan on what will need to be done. In my second installment I wrote about the first considerations of what I need the tooling to be able to do. In this part I am going to discuss a few of the things I have done to come to the shortlist and what will I do as a Proof of Concept for the tools.

Load generator

First of all, let’s get the easy part out of the way. We will need something to generate a (functional) load on the servers. That part i consider relatively easy, no big tools are needed for this since we have an extremely powerful opensource tool at our fingertips: Apache’s JMeter. JMeter The load will have to be generated based on both HTTP traffic and client/server traffic, neither of which should pose a problem for JMeter. The most difficult part for load generation is getting the numbers out of the system, e.g. figuring out what the average and peak load is on the system. For this we have thrown some lines out to application managers to figure out.

Long list

The longlist I started out with was not just any list, it was a set of several lists. Out of this initial set I picked a bunch to actually play around with a bit more. Some gave me fun new insights, some disappointed me from the beginning, just by reading the sites or white papers.

The list of tools I initially looked at somewhat seriously was the following:

telerik-logo

Logo_froglogic

SmartBear-New-Logo_RBG

header-logo-borland

1350141391

Original_Software_logo

autoit-logo

thoughtworks-logo

logo-neotys-top

 

Shortlist

Quite a few of the tools I installed, just to see how they work and integrate with developer tools like Eclipse and Microsoft Visual Studio Express. The majority of the more expensive tools barely integrate at all, since I would need to have a full version of Visual Studio rather than the Express version. That is a full disqualifier for me in this phase.

Another strong disqualifier is if the tool simply refuses to run on a Windows XP Professional environment, such as Microsoft Visual Studio Test Professional. Within this company the majority of machines are still running Windows XP or XP Pro, so the tools need to work perfectly in that environment. Interestingly the only tool that flat-out refuses to be installed on it is a Microsoft own tool :).

After having considered the needs for the tool in the short term and possibly longer run two tools jumped out big time: Borland SilkTest and Sikuli.

What have been (some of) the disqualifiers for the other tools I looked at:

  • availability of a downloadable and fully usable demo version, some tools have no demo version available or the demo is locked off.
  • support of SAP for possible future use, the organisation is looking at a long road ahead of SAP upgrades and patches, so automated test support would be a welcome helping hand
  • possibility to use the tool for more than just performance or load testing, for example for pure functional test automation
  • organizational fit moving forward, e.g.
    • will the less technical people within the organisation be capable of using this tool for future runs of the tests built for this particular project?
    • will this tool be capable of supporting upcoming projects in both functional and non-functional tests?
    • is the learning curve for the internal users not too steep (or, how much programming is actually needed)
  • price, is the price something that fits within the project budget and does the price make sense in relation to the project and capabilities of the tool

PoC

With this very short list of two tools a Proof of Concept will be made to see how the applications deal with several situations I will be running into during the performance tests.

One of the main parts to test is whether or not the tool is accurate enough in measuring and reading the state of the application under test. Since the application under test is two fold: a web-application and a remote desktop application.

The webapplication, as stated in the previous post, will not really be the difficult one to test. The remote desktop application however is more challenging to test. The application runs on a Citrix server and thus the object ID’s are not visible to the test automation tooling. The second  outcome of the PoC should be to see how well the tooling deals with the lack of object ID’s and thus with navigating the application based on other pointers. For Sikuli the challenge will be different resolutions, for SilkTest I will be focusing on finding a way other than navigating by screen coordinates.

Automating SAP to create load from an RFC port

In my current assignment I am tasked with coordinating the testing of the integration of several retail systems, basically making them work together logically and effectively. Part of the work is  oriented towards load and performance testing of these integrated systems.

What is being done is that SAP Retail systems need to communicate with Locus WMS, since the version of SAP currently running at the customer cannot deal with anything but IDocs a message broker has been setup in between SAP and Locus to translate the IDocs into XML and vice versa. The IDocs are served to the message broker via SAP’s default RFC port, the broker pulls the documents out of SAP, translates them and sends them off to Locus to be picked up and processed. This is a simplification of how it truly works, since it is only meant to help set the scene.

Generating IDoc load

In order to build up load in a structured, guided way from SAP there are a few ideas of what can be done. My initial hopes, were to push IDocs from a load generator to the message broker. This would be the easiest way in which to control the flow of data towards the broker and thus the easiest way to make sure we are fully in control of how busy the broker is. Alas, when talking to the guys behind the broker interfaces it turned out that this method would not work for the setup used. The only way the broker would actually do something with the IDocs was if it could pull them from the SAP RFC port, pushing to the broker would not work, since the RFC receiving end of the broker is not listening, it is pulling.

Alternatively sending data off into the message queue would fill up the MQ, but not help with getting the messages pushed through the Broker, again, due to the specific setup of the Enterprise Service Bus which contains the broker interfaces.

Spike testSo alternatives needed to be found. One obvious alternative is setup a transaction in SAP which generates a boat-load of IDocs and sends the to the RFC port in one big bulk. This would generate a spike, such as shown in this image, rather than a guided load. In other words, this is not what we want for this test either. It might be a useful step for during a spike test, however the first tests to be completed are the normal, expected load tests.

The search for altneratives continued. At my customer, not a lot of automation tools were available, especially not for SAP systems. One tool however has been purchased a while ago and apparently is actively used: WinShuttle

Winshuttle seems to be able to generate the required load, based on Excel input, the main issue with Winshuttle however, was the lack of available licenses. There are two licenses available and both are single use licenses. This meant I would have to find a way to hijack one of the PC’s it was installed on, script my way through it and run the tests in a very timeboxed manner. In other words, not really a solution to the problem.

I then decided to look at this from a whole different point of view: what can I use to make SAP execute a bunch of transactions, is freely available and flexible enough to also monitor what is happening on several sides of the message broker? The answer that came to me was not quite what I had expected: AutoIt.

SAP-main-screen-side-by-side
Starting SAP from AutoIt was simple, running through the application and manipulating SAP however was a bit less intuitive.
In this screenshot two SAP screens are put side by side, the left-hand side is what the userinterface in SAP looks like to the end user. The right-hand side is how AutoIt sees the screen, e.g. a big blob of nothingness.

SAP-AutoItScreenInfo3

To be a bit more specific, here’s what AutoIt can tell us about the SAP toolbar:

In other words, AutoIt sees the entire toolbar as one object, with one exception, the edit box for transactions. This box has a very easy and intuitive name: Class: Edit Instance: 1, making it easy to ensure the focus on this box can be easily set and thus the transaction being started to upload files.

Since the main screen of SAP is a blind box to AutoIt we had to resort to a very sloppy way of working, using the TAB button to navigate through the screen, resulting in code roughly looking like this:

Send("ZWBESTUPL{ENTER}")
 If WinWaitActive("Bestellingen (winkel) aanmaken vanuit CSV-bestand") Then
    _log('Successfully selected the Bestellingen aanmaken vanuit CSV-bestand transactie')
    Send("{TAB 3}")
    Send("{SPACE}")
    Send("+{TAB 3}")
 Else
    _log('Something went wrong. Could not get to the Bestellingen aanmaken vanuit CSV transactie')
    Exit
 EndIf

load-graph The resulting load ramp up was a linear rampup of IDocs being generated and sent to the SAP RFC port, where they were picked up by the Message Broker and subsequently tranformed and sent back to the Locus system, where the load turned out to be quite on the high side.

All in all this was a fun excercise in automating SAP to do something it is absolutely not meant to do with a tool not built nor designed to do what it did. In other words, it was wonderful to be able to abuse a bunch of tools and achieve a very clear and convincing result!

Test automation on SAP, is it really that much different?

SAP logo This year I got to know SAP fairly intimately, looking at it and into it from a test automation perspective, inventorising the possibilities and opportunities of automated testing of a (huge) SAP implementation. During this time I ran into a fair amount of SAP related people, ranging from SAP consultants and sales people to ABAP-developers, HP sales people and SAP preferred suppliers. They all are making it seem as though SAP development and testing is a different world, nothing to do with the “normal” software development world. In my view this is wrong, SAP is just software. Yes, it has a bunch of particularities which you do not get in so many other packages, but in terms of the actual functionality it is fairly comparable to Siebel and Oracle (no, I am NOT saying it’s the same, I am merely saying it is comparable). With neither Oracle nor Siebel this almost religious separatism exists, yet they too are bound by the laws of business process models, transaction codes and what not. So how come SAP is seen as so special and the others are not? Is SAP special? SAP TAO & HP Quality CenterWhen you start talking about test automation and SAP the first things that pop up are some SAP proprietary names such as CATT, eCATT and SAP TAO. Fortunately SAP themselves recommend against the use of either CATT or eCATT, so let’s dismiss these right here and now, they are tools that once were somewhat helpful but now should be considered redundant for most SAP implementations. SAP TAO however is of a different breed. SAP TAO is pushed by SAP as being the solution to use when trying to automate your testing. One minor issue with SAP TAO however is that it does not really automate anything on its own, you invariably need HP Quality Center (HPQC) and Quick Test Professional (QTP) with it. HP tooling has some tailor-made solutions to integrate well with SAP TAO and more specifically with the SAP Solution Manager. The setup as proposed in this picture is the ideal picture as SAP would like to envision and implement a SAP testing solution. However, not all organisations have Solution Manager up and running for anything other than transport and low level reporting, nor do all organisations have the budget for the HP tool set. When working with SAP TAO effectively and efficiently, the Business Blueprint, the description of all business processes as used by the organisation with the SAP systems, should be residing in the SAP Solution Manager. This blueprint should be maintained carefully and always be up to date. When changes to the system are made, either by updates to the system or by customizations in ABAP, these changes should be visible in the Solution Manager, ensuring the SAP Solution Manager Business Process Change Analyzer can identify which processes have changed and based on this impact analysis propose tests within HP Quality Center to be executed. With SAP TAO the testers can “automate” the tests, which effectively means record the steps. SAP TAO then adds some secret sauce by cutting longer scripts up into maintainable and reusable chunks. These scripts will then be sent from SAP TAO into HPQC, where they can be associated with functional test descriptions. When a tester now wants to run one of the automated tests, or for that matter wants to run the entire automated suite,  HPQC is used again to trigger the scripts, which get executed with QTP. In other words, the actual testdriver is QTP, not SAP TAO. When starting up a SAP GUI instance and analyzing it with something like UISpy or some other tool which can show the objects on a screen, the fields and buttons are barely visible and not really open to test automation. Yet it is possible. If SAP is configured to enable scripting, the UI objects become accessible and thus the GUI is scriptable with any tool of your choice. The moment this little flag has been set, a whole new world opens up in the GUI, it’s all of a sudden open, the fields, screens and buttons all have an ID and can be hooked into by a driver of your choice. Effectively what the enable scripting setting does, is ensuring non of the huge, expensive tools mentioned above are needed, it is possible to run through the application with any driver you want. The main thing needed in order to properly and solidly automate testing in SAP now, is a well grounded knowledge of the Business Processes the implementation is supporting (or driving).  This is no different than what is needed when automating SAP with SAP TAO. The benefits of having the option to choose your own drivers, your own programming language and your own reporting framework are huge. If SAP is merely in the organisation to support the business processes and software developers within the organisation are writing their own code in Erlang, C++, C#, Java, Ruby, Python or whatever else you can imagine, the testsuite for SAP can be in that same language. Having the automated testsuite in a well supported language rather than just in QTP’s own VBScript, ensures a larger possible support base for the automated tests. It enables easy integration of home-built software with the SAP systems since all tests can be built in one language and in an end-to-end setup, again supported by the organisation’s own development group. The SAP TAO and HPQC setup do have some benefits of course. First of all, there is a huge corporate support for both HP and SAP software products. But more importantly, there are some technical benefits of using SAP TAO, if the environment is setup properly. As mentioned above, there is this tool called the Business Process Change Analyser, or BPCA, which can help extract transaction based changes from a transport and help the tester decide, based on these changes, which test scenarios need to be run to effectively cover the business processes (or mainly the transactions associated both directly and indirectly to the transport). Next to that there is the benefit of using HPQC, I can hardly believe that I am saying this, since I am personally not a big fan of the HPQC suite, however the reporting possibilities and capabilities within HPQC are close to limitless. This means that it is possible to generate excellent reports, automatically, for both management level execs and for the business analysts and ABAP-specialists, on each test run without having to think about it. Having the full benefits of this setup however comes at a cost, a fairly sizable cost. The licensing for HPQC, QTP and SAP TAO or not to be ignored for starters. A hidden cost lays within the organisation, as stated, for the BPCA to do anything, Solution Manager needs to be utilized fully, the Blueprint needs to be ready and up to date, more over, it needs to be well maintained to ensure it remains the “Single Source of Truth” (as SAP coined it). So, to answer the initial question: Is SAP special? It is, as a business process tool, definitely special, strong and extremely versatile. When looking at SAP as a system that requires testing and test automation however, I am not convinced it is special, it’s just software, which is open for testautomation with a range of drivers, one of these drivers might be QTP. If you do indeed choose to go for QTP with a SAP system, have a look into SAP TAO. However, do not feel that it is the only one out there which can effectively and efficiently be used for SAP test automation. All the others claiming they can, probably indeed can just as well as SAP TAO with QTP. In the end it is all about how you use and abuse a tool and whether you use QTP, White or Panaya, they all in the end merely function as a driver, it is the code the testers build which matters!

The cost of test automation

Over the past few posts I have written a lot about test automation, however one very important subject I have left out thus far. What is the actual cost of test automation? How do you calculate the cost of test automation, how do you compare this cost to the overall costs of testing? In other words, how do you get to the return on investment people are looking for? First thing that needs to be covered if you want to know and understand the costs of test automation is a CLEAR understanding of the goal. Typically there are three possible goals:Cost, quality or time to market?

  1. reduce the cost of testing
  2. reduce the time spent on testing
  3. improve the quality of the software

These three have a direct relation with each other, and thus each of them also has a direct impact on the other two depending on which one you take as your main focus. In the next few paragraphs I will discuss the impact of picking one of the three as a goal.

Reduce the cost of testing

When putting the focus of your test automation on reducing the overall cost of testing you set yourself up for a long road ahead. There generally is an initial investment needed for test automation to become cost reducing. Put simple, you need to go through the initial implementation of a tool or framework. Get to know this tool well and ensure that the testers who need to work with the tool all know and understand how to work with it as well. If going for a big commercial tool there is of course the investment in purchasing the license, which often ranges between 5.000 and 10.000 euros per seat (floating or dedicated). Assuming more than one test engineer will need to be using the tool concurrently, you will always need more than one license. This license cost needs to be earned back by an overall reduction in testing costs (since that is your goal). An average investment graph will look something like this when working with a commercial tool: Investment of test automation with a commercial tool

The initial investment is high, this is the cost of the licenses. At that point there is nothing yet, no tests have been automated yet and you have already spent a small fortune. The costs will not drop immediately, since after the purchase the tool needs to be installed, people need to be trained etc. Next to it, all the while no tests have been automated, thus the cost is high, but the return is zero. Once the training process has been finilised and the implementation of the automated tests has started the cost line will slowly drop to a stable flatline. The flatline will be the running cost of test automation, which includes the cost of maintenance of the tool and the testscripts and of course the cost of running tests and reviewing and interpreting the reports.

A particular post in a LinkedIn group with the ominous question “Who’s afraid of test automation”, one of the more disturbing responses was as follows (and I am quoting from the group literally, with a changed name however):

Q: Who’s afraid of test automation?

A: Anyone with headcount. What would it look like if all of the testing is done by machine and there is only one person left in the organization?
Respectfully, Louise, PhD

The idea that test automation will take away testers’ work completely and as such will reduce the running costs of a test team drastically is a common misconception which takes too much time and effort right now to address. But in short, Test automation may reduce some of the workload, but will not be able to reduce your cost of testing by removing all of the testers, nor should this ever be your objective.

In my next blog post I will continue this story, then with the focus on “Reduce the time spent on testing”

The difference for test automation between cutting edge and legacy software

Within one of the LinkedIn groups (sorry, you need to be a member of the “QA Automation Architect” group to be able to read it fully) we started talking about the difference the state of project or product can make for test automation. In this post I will make a distinction between 2 states: new where no code has been written yet and existing  where application code has been written, but no test automation has been implemented.

Cutting edge

New So when creating a totally new product, life for the testers can be made easier by design, that at least is the thought. This does imply that testers, and not just the “manual” testers but all testers, including automation testers if these are a separate breed as some people seem to think, need to actively participate in the requirements phase of a product. With actively participating I do not mean to imply that they are normally not participating, I mean they need to look a bit further than just at what to test, is it testable etc.

They should also use their insights and ideas to help both product owners and software developers to understand what are the things that might make life easier for testing this new product.

When for example building a new web application, they might consider adding a simple REST api to the application, which in production can be closed off based on IP or firewall rules or something like that. A simple REST-API will make life a lot easier when creating your automated tests.

Another thing to make life easy might be ensuring clear and logical naming conventions to be used for all page object in order for the automation to use the Page-Object-Model. Not only is using solid naming conventions good for automation, it also makes maintenance on the application itself easier, since all objects are identifiable by their unique ID.

Legacy

How is existing code different from non-existent, other than that one is already in production and the other has to be created? As far as test automation is concerned, especially when talking about legacy software, it may turn out to be a lot more difficult to find proper hooks into the application for solid automation other than on the labels of buttons or fields.

When you have a fairly recent application it may be a website or a desktop app, both have the possibility that there are some sorts of ID’s for all objects. However when talking about true legacy software, such as 15 year old Delphi, it is quite unlikely the developers used WinForms, Win32 or SWT. Not having hooks like that into the application can result in having to scrape the UI for object labels, which is fine when testing one particular language, but if your software was localized things can get even more complicated.

Getting consensus within the technology group about new software is one thing, getting a “non-functional”, non-business related change about in existing software however is a whole different thing.

As long as the code is still “alive”, e.g. new features are still being added, bugs are being fixed and in general there are still developers working on the application, there is hope of getting some more “automatability” in the code.

First of all, while fixing bugs old code is touched, adjusted and retested, this is always an opening to talk to the developers resolving the issue about adding a small bit of extra “sauce” to make it easier to add this particular thing to the automated testing suite to ensure chances of recurrence are minimized, of course by fixing the bug you hope to completely obliterate this particular issue but it might cause new damage elsewhere in the application. So while talking to the developer about this function, try to convince him/her that adding a bit of extra to test not only for the fix of this issue, but also to verify the surrounding features.

While new features are added, this can be treated as “new code”, as long as you manage to get agreement on adding identifiers or a separate layer in these features to make test automation at least easier. If you achieve this, you are quite close to closing the majority of the gap. Refactoring is an excellent opportunity to again make minor changes in the application enabling test automation at a different level.

How do you get “automatability” in your specs?

Assuming you want to get your  product easy to automate and thus want to make sure it is thought through, how to get it in the specifications? And more importantly, how do you get it in there without adding things like:

  • unnecessary workload
  • unneeded and unwanted features
  • potential security holes
  • un-maintained code

Enterprise Architecture Layers with a "hidden test automation layer"One of the ways to go about it is by, in collaboration with the developers, enforcing a coding standard in which you ensure all objects receive an ID. Regardless of whether it is desktop or web based, most automation tools are looking for a hook into the UI, if there is one, and one of the nicest ways of doing that is simply by using the ID.

Alternatively you can have a “layer” put right underneath the UI, ensuring you can bypass the cumbersome UI while automating your tests. One of the issues with this option however, can be that you add “hidden” code which gets forgotten easily. It also is a potential risk for the security of your application, since you basically enable a man-in-the-middle hole.

If this path is taken, ensure that this “feature” does not end up being an opening for malicious code to reach your data. A relatively safe solution for this would be to put some (extra) form of authentication in the layer.

There probably are more options you can investigate, the two I mention above are fairly harmless and yet can make life in test automation a lot easier and predictable.

In the end, no matter which way you go, as long as you get both developers and product owners on board in working towards a higher “automatability” of the code life for you as a test engineer could become a lot more fun.

I am not very impressed with theological arguments whatever they may be used to support. Such arguments have often been found unsatisfactory in the past.

Alan  Turing