Tag Archives: load & performance testing

Standard

Posted by

Posted on

November 26, 2014

Posted under

Load & Performance testing

Comments

Leave a comment

JMeter – private remote servers or a service?

Over the past year I have spent a lot of time in close collaboration with Apache JMeter. I used JMeter as a standalone platform, in collaboration with several tools out there and with my own servers. Over the next few posts I will elaborate a bit more on what I have learned from working with tools like BlazeMeter, Loadosophia as well as working with your own remote servers and Jmeter. On top of that I will share some things I consider good practices to use when working with JMeter and building elaborate test scenarios in it.

JMeter and remote servers

Apache JMeterJMeter is great as a stand-alone load generator, however the moment you need to generate a substantial load or a distributed load, you will require the use of more servers. There are several possibilities to add external servers to JMeter. You can use a loadgeneration platform such as BlazeMeter and Flood.io or you can choose to use your own machines (Disclaimer: I have only worked with Flood and BlazeMeter as services, so I only mention these two. There may very well be other services out there I am not (yet) aware of).

Why use your own remote servers?

Virtual Private ServerAlthough there are several commercial options to run JMeter tests from a cloud-platform, it might be useful to have your own servers running. Your own machines, for example Virtual Private Servers, will give you at least the following advantages, please bear in mind these are the main reasons for me to run private servers. It may very well be that you have different reasons for doing the same (I’d be happy to hear them btw!):

  • Availability within firewalls: Not all applications are available outside of firewalls. For quite a lot of companies it is not an option to allow external addresses, especially cloud services, within their firewalled environment. When this is the case in your organisation it can be very useful to have your own load generator hosted inside the firewall.
  • Geographical location: not each and every country has platforms available for load generation. Although platforms like Blazemeter have a lot of different locations available, it may very well be that your particular country is not available through a serviceprovider. However getting your own, hosted, machines within a country is generally not too difficult to do. Keep in mind though whenever you want to use something like a Virtual Private Server to verify with the hosting company whether they allow load generation from within their network!
  • Control: a fairly straightforward reason is in order for you to have full, unlimited control over the load generating servers. If you have your own (virtual) servers running you can easily adjust settings, properties and other things in order to make it fully match your (customers) needs.

Why use a service?

BlazeMeterflood.io

Even though there are quite some reasons for using your own machines, the advantages of using a service like BlazeMeter or Flood.io  are plenty, I will just highlight the few that for me have tipped the scales several times in favor of a service.

  • Maintenance: Setting up your own servers is time consuming and therefore not cheap. These servers need maintenance even though you quite likely do not constantly need them. A Service is just there. You buy a subscription, be it with a time constraint (monthly) or a load constraint (max amount of concurrent users) or something like that, but the service is simply at your fingertips. The moment you need it, you have it.
  • Multiple locations: the JMeter based services generally make use of the Amazon AWS cloud, thus giving you, the user, an immense amount of servers at your disposal. With this comes a huge worldwide distribution, which can be very advantageous for big world-wide used applications.
  • Support: both mentioned platforms have a very solid support base for their customers. If you have questions, if your tests seem to not run properly or you simply have issues getting something done, there generally is a good support platform available. You can use forums or the actual helpdesk, but either way, there is a good, solid, commercial support-base to help resolve your issues.

In the end, I choose per assignment what best fits my needs. Sometimes I use the VPS, sometimes I use a service. It is important you at least think about what best suits your needs for the test you are about to build and execute.

Standard

Posted by

Posted on

May 2, 2013

Posted under

Load & Performance testing, Test automation

Comments

3 Comments

Selecting performance test tooling – Part 3

A birthday post… and no, the story has nothing to do with my birthday, it’s just that today is my birthday.

This post became a bit longer than I initially intended so here are some links to jump directly to the specific content:

Some challenges in the PoC…

Following up on my previous posts (here and here) on this topic while executing the Proof of Concept I have run into some interesting challenges.

For starters, I am convinced I started the PoC the wrong way around; I started off with implementing some things in Sikuli rather than in SilkTest. Since SilkTest is, to me, less intuitive than Sikuli, since SilkTest tries to encapsulate a lot of different automation approaches in one tool, whereas Sikuli is focussed on one methodology only, I should have started with that one. However, I didn’t and there is nothing I can do about that anymore now.

Secondly, the application we are about to put to the test is an application served from a Citrix platform, in other words, it is a remote application. The charter for this project is simple: Measure the performance of the application as the user would experience it. In other words, measure its performance via the RDP tunnel and not directly on the Citrix machine.

The setup is basically as shown in this image (simplified of course)

Citrix simplified setup

Simplified picture of the Citrix application setup

Sikuli Sikuli Script Logo

For those not farmiliar with Sikuli, here’s what they say about themselves:

Sikuli Script automates anything you see on the screen. It uses image recognition to identify and control GUI components. It is useful when there is no easy access to a GUI’s internal or source code.

In other words, Sikuli is fully based on image recognition and pattern recognition rather than following the industry standard Object Model.

The good, the bad and the ugly.

Stepping away from the Object Model has some advantages, especially in this application setup, but I will get to that when discussing the Borland setup.

The good

Considering this is a Proof of Concept I have simply taken Sikuli out of the box, using Sikuli-IDE. The IDE works nice, simple and intuitive. It was very easy to start the RDP application and login without using any screenshots. The basic use of Sikuli is very simple and intuitive. Scripting in it is simple and logica, at least if you have a basic understanding of other scripting languages and/or programming.

Functionally stepping through the application was easy, just a few small screenshots were needed to load reports and verify that the report indeed is loaded successfully. In other words, the ease of use is excellent!

The bad and the ugly

I am mashing the bad and the ugly into one big pile since they are closely connected.

Sikuli-Code-SnippetThe first thing I disliked a lot is that Sikuli is 100% depending on Java 6, try running it on 7 and you have a problem (as in, it simply doesn’t work).

Another bad part of Sikuli is that even if I wanted to, I cannot add Object ID’s. This means that if I want to verify the existence of something, it needs to be done with screencaps and recognition thereof. Which leads me to the ugly. Screencaps are not the nicest way to identify objects, in fact they are ugly and not friendly to use, since objects can occur, in a similar look and feel, several times on one screen. This results so now and again in the wrong button being clicked. It may look the same to Sikuli, but it is not the same functionally.

On top of that, I am now saving images in source-control (GIT) which I am not in favor of. Why would I want binary files in source control? I cannot do a diff on them anyway.

SilkTest Borland logo

I have known Borland as a company for a long time yet in the past 10 years have not really worked with any of their tools. A short summary of how they see themselves:

With Silk Test, there’s no need to understand coding so even non-technical people like your business analysts can build tests and get fully involved. This 13.5 release also breaks new ground by working with all the latest browsers, so a single script is all you need.

Well, there are some issues with that statement of course, cause a single script is always doomed to fail in the most horrid ways imaginable, but still, SilkTest is a nice tool to work with.

The good

The reason for looking at SilkTest was because I would like to have a tool now which is future proof for the organisatio. In other words, will this tool support further test automation on the end-to-end chains within this large organisation. One really important qualifier for that is solid SAP support. My Proof of concept on SilkTest started off looking into SAP support. The way Silk handles SAP I can simply summarize with one word: good. Out of the box it managed to select the correct SAP instance from the system selection popup, login without issues and after a few attempts execute a bunch of transactions. In other words, I was happily surprised! Most test automation applications I had on the longlist have serious issues in dealing with SAP.

The bad and the ugly

UISpy view of the applicationThe not so nice side of SilkTest in my opinion is that the recorded code is somewhat ugly, if not really ugly and not very friendly to read and through that probably also to maintain. This however is just a minor nuisance compared to the next issue.

Since the application under test is being served through an RDP tunnel I have no access to the object ID’s. In other words, it is difficult to recognize objects on the application. In SilkTest it is not merely difficult, it is close to impossible. The only runnable way to do so I found is to record the tests based on the screen coordinates and then manually add assertions all over the place. However since SilkTest doesn’t see what it is trying to test, getting the assertions in is really hard. What do you put the assertion on? There is no object to verify.

In other words, this is a disqualifier for SilkTest in this context.

Standard

Posted by

Posted on

April 29, 2013

Posted under

Load & Performance testing, Test automation

Comments

5 Comments

Selecting performance test tooling – Part 2

In my first installment I wrote about how I got the requirements together. Based on those I wrote a plan on what will need to be done. In my second installment I wrote about the first considerations of what I need the tooling to be able to do. In this part I am going to discuss a few of the things I have done to come to the shortlist and what will I do as a Proof of Concept for the tools.

Load generator

First of all, let’s get the easy part out of the way. We will need something to generate a (functional) load on the servers. That part i consider relatively easy, no big tools are needed for this since we have an extremely powerful opensource tool at our fingertips: Apache’s JMeter. JMeter The load will have to be generated based on both HTTP traffic and client/server traffic, neither of which should pose a problem for JMeter. The most difficult part for load generation is getting the numbers out of the system, e.g. figuring out what the average and peak load is on the system. For this we have thrown some lines out to application managers to figure out.

Long list

The longlist I started out with was not just any list, it was a set of several lists. Out of this initial set I picked a bunch to actually play around with a bit more. Some gave me fun new insights, some disappointed me from the beginning, just by reading the sites or white papers.

The list of tools I initially looked at somewhat seriously was the following:

telerik-logo

Logo_froglogic

SmartBear-New-Logo_RBG

header-logo-borland

1350141391

Original_Software_logo

autoit-logo

thoughtworks-logo

logo-neotys-top

 

Shortlist

Quite a few of the tools I installed, just to see how they work and integrate with developer tools like Eclipse and Microsoft Visual Studio Express. The majority of the more expensive tools barely integrate at all, since I would need to have a full version of Visual Studio rather than the Express version. That is a full disqualifier for me in this phase.

Another strong disqualifier is if the tool simply refuses to run on a Windows XP Professional environment, such as Microsoft Visual Studio Test Professional. Within this company the majority of machines are still running Windows XP or XP Pro, so the tools need to work perfectly in that environment. Interestingly the only tool that flat-out refuses to be installed on it is a Microsoft own tool :).

After having considered the needs for the tool in the short term and possibly longer run two tools jumped out big time: Borland SilkTest and Sikuli.

What have been (some of) the disqualifiers for the other tools I looked at:

  • availability of a downloadable and fully usable demo version, some tools have no demo version available or the demo is locked off.
  • support of SAP for possible future use, the organisation is looking at a long road ahead of SAP upgrades and patches, so automated test support would be a welcome helping hand
  • possibility to use the tool for more than just performance or load testing, for example for pure functional test automation
  • organizational fit moving forward, e.g.
    • will the less technical people within the organisation be capable of using this tool for future runs of the tests built for this particular project?
    • will this tool be capable of supporting upcoming projects in both functional and non-functional tests?
    • is the learning curve for the internal users not too steep (or, how much programming is actually needed)
  • price, is the price something that fits within the project budget and does the price make sense in relation to the project and capabilities of the tool

PoC

With this very short list of two tools a Proof of Concept will be made to see how the applications deal with several situations I will be running into during the performance tests.

One of the main parts to test is whether or not the tool is accurate enough in measuring and reading the state of the application under test. Since the application under test is two fold: a web-application and a remote desktop application.

The webapplication, as stated in the previous post, will not really be the difficult one to test. The remote desktop application however is more challenging to test. The application runs on a Citrix server and thus the object ID’s are not visible to the test automation tooling. The second  outcome of the PoC should be to see how well the tooling deals with the lack of object ID’s and thus with navigating the application based on other pointers. For Sikuli the challenge will be different resolutions, for SilkTest I will be focusing on finding a way other than navigating by screen coordinates.

Standard

Posted by

Posted on

April 11, 2013

Posted under

Load & Performance testing

Comments

3 Comments

Selecting performance test tooling – Part 1

How do you come to a logical and effective performance test tool set? Yes, I say tool set since quite often just one tool will not suffice.

As could be read in my previous post I am currently heavily involved in performance testing for a large Dutch retail organization which is merging two large organisations into one. For the ERP system they both use I have written a concise Performance testplan. Now the time to execute has come.

In order to execute the performance tests on the ERP system, which is both desktop application (WPF) and web-based I will need a solid tool to

  1. generate load on the servers by emulating functional  behaviours
  2. functionally walk through the actual desktop and web-applications to measure the true application performance
Performance testing

Performance testing

Generating load is not the biggest challenge, quite a few tools are capable of doing that, especially since there is a website we can push the load through. The functional walk-through is going to be a bit less simple to create, especially since I have to recreate the same scenarios twice; once for the web application and once for the desktop application.

To complicate things even more, I want the functional walk-through  and thus the real measurements, to be done by the same tool. This way I can make sure the measurements are reporting on the same thing in the same way and thus can be compared to one-another to some extent.

Long list

First off in looking for the right tools I started setting up a long list, or actually, I Googled for a longlist…
This search resulted in several useful pages I checked out:

Just based on previous experience, tool knowledge etc. I dismissed a set of the tools and ended up with a nice set of  possible tools.
Next step, which of these tools actually fit with the organisation and application landscape? I have specific needs at the moment for this tool, however I also need to keep the future needs of my customer in mind. In other words, I need a tool that is a) good for me now and b) good enough for the foreseeable future to be worth the investment (in license and time or just time, either way quite an investment).

What will the tool need to support?

  • WPF/Win32/WinForms
  • Java (Swing) UI’s
  • Oracle
  • SAP GUI
  • Web applications of all sizes and shapes
  • RFC communications
  • SOAP
  • Tibco

The search has begun.
In the next installment of this series I will tell more about what the shortlist has become, what I tested on the applications and how I have come to a decision which application seems the best fit for this environment.

Standard

Posted by

Posted on

February 8, 2013

Posted under

Software testing, Test automation

Comments

2 Comments

Automating SAP to create load from an RFC port

In my current assignment I am tasked with coordinating the testing of the integration of several retail systems, basically making them work together logically and effectively. Part of the work is  oriented towards load and performance testing of these integrated systems.

What is being done is that SAP Retail systems need to communicate with Locus WMS, since the version of SAP currently running at the customer cannot deal with anything but IDocs a message broker has been setup in between SAP and Locus to translate the IDocs into XML and vice versa. The IDocs are served to the message broker via SAP’s default RFC port, the broker pulls the documents out of SAP, translates them and sends them off to Locus to be picked up and processed. This is a simplification of how it truly works, since it is only meant to help set the scene.

Generating IDoc load

In order to build up load in a structured, guided way from SAP there are a few ideas of what can be done. My initial hopes, were to push IDocs from a load generator to the message broker. This would be the easiest way in which to control the flow of data towards the broker and thus the easiest way to make sure we are fully in control of how busy the broker is. Alas, when talking to the guys behind the broker interfaces it turned out that this method would not work for the setup used. The only way the broker would actually do something with the IDocs was if it could pull them from the SAP RFC port, pushing to the broker would not work, since the RFC receiving end of the broker is not listening, it is pulling.

Alternatively sending data off into the message queue would fill up the MQ, but not help with getting the messages pushed through the Broker, again, due to the specific setup of the Enterprise Service Bus which contains the broker interfaces.

Spike testSo alternatives needed to be found. One obvious alternative is setup a transaction in SAP which generates a boat-load of IDocs and sends the to the RFC port in one big bulk. This would generate a spike, such as shown in this image, rather than a guided load. In other words, this is not what we want for this test either. It might be a useful step for during a spike test, however the first tests to be completed are the normal, expected load tests.

The search for altneratives continued. At my customer, not a lot of automation tools were available, especially not for SAP systems. One tool however has been purchased a while ago and apparently is actively used: WinShuttle

Winshuttle seems to be able to generate the required load, based on Excel input, the main issue with Winshuttle however, was the lack of available licenses. There are two licenses available and both are single use licenses. This meant I would have to find a way to hijack one of the PC’s it was installed on, script my way through it and run the tests in a very timeboxed manner. In other words, not really a solution to the problem.

I then decided to look at this from a whole different point of view: what can I use to make SAP execute a bunch of transactions, is freely available and flexible enough to also monitor what is happening on several sides of the message broker? The answer that came to me was not quite what I had expected: AutoIt.

SAP-main-screen-side-by-side
Starting SAP from AutoIt was simple, running through the application and manipulating SAP however was a bit less intuitive.
In this screenshot two SAP screens are put side by side, the left-hand side is what the userinterface in SAP looks like to the end user. The right-hand side is how AutoIt sees the screen, e.g. a big blob of nothingness.

SAP-AutoItScreenInfo3

To be a bit more specific, here’s what AutoIt can tell us about the SAP toolbar:

In other words, AutoIt sees the entire toolbar as one object, with one exception, the edit box for transactions. This box has a very easy and intuitive name: Class: Edit Instance: 1, making it easy to ensure the focus on this box can be easily set and thus the transaction being started to upload files.

Since the main screen of SAP is a blind box to AutoIt we had to resort to a very sloppy way of working, using the TAB button to navigate through the screen, resulting in code roughly looking like this:

Send("ZWBESTUPL{ENTER}")
 If WinWaitActive("Bestellingen (winkel) aanmaken vanuit CSV-bestand") Then
    _log('Successfully selected the Bestellingen aanmaken vanuit CSV-bestand transactie')
    Send("{TAB 3}")
    Send("{SPACE}")
    Send("+{TAB 3}")
 Else
    _log('Something went wrong. Could not get to the Bestellingen aanmaken vanuit CSV transactie')
    Exit
 EndIf

load-graph The resulting load ramp up was a linear rampup of IDocs being generated and sent to the SAP RFC port, where they were picked up by the Message Broker and subsequently tranformed and sent back to the Locus system, where the load turned out to be quite on the high side.

All in all this was a fun excercise in automating SAP to do something it is absolutely not meant to do with a tool not built nor designed to do what it did. In other words, it was wonderful to be able to abuse a bunch of tools and achieve a very clear and convincing result!