JMeter and WebDriver – Why would you want to combine them?

3 Reasons to combine JMeter and WebDriver in performance testing a webapplication


What is JMeter good at?

The Apache Foundation states the following about JMeter:Apache JMeter

The Apache JMeter™ desktop application is open source software, a 100% pure Java application designed to load test functional behavior and measure performance. It was originally designed for testing Web Applications but has since expanded to other test functions.

So in other words, it’s a tool you can use to generate functional load on an application or a platform. This also immediately describes what it is good at: generating load. Yes if implemented well you not only generate dumb-load but also hit the functional application layers with the tool. But the basic function of JMeter is aimed at generating load and measuring the (server) response times during this load.

What does JMeter NOT do?

Despite being capable of generating functional load, JMeter does not render pages nor is it very well equipped to execute embedded JavaScript (it is simply not equipped to do that actually), therefore JMeter will not tell you anything (out of the box) about the render times of pages. Especially not about render times when the server is heavily overloaded by your scripts!

What is WebDriver good at?

SeleniumHQ gives a wonderful description of what Selenium (or nowadays WebDriver) does:
Selenium Logo

Selenium automates browsers. That’s it! What you do with that power is entirely up to you. Primarily, it is for automating web applications for testing purposes, but is certainly not limited to just that.

In short, what Webdriver does is just about anything that happens within your browser. It does render pages, it does execute JavaScript, it retrieves the pages as if an actual human was clicking on a website. So for fully functional automated testing (or checking to stick with the more correct terminology) WebDriver is perfect.

What does WebDriver NOT do?

Well, it is not quite good at generating load. Since WebDriver basically requires a browser (yes, it is possible to run it headless of course) it is very difficult to generate multiple (virtual) users. That would require a bunch of browsers to start up, when talking about 10 users that may seem feasible, however when talking about generating real load (say several 1000’s concurrent users) a bunch of browsers becomes a lot more difficult to arrange.

Why combine them then?

The logical question then indeed is, why would you combine them? Below I have set out 3 clear reasons why combining JMeter and WebDriver scripts can be an excellent idea.

  • Impact of server-side load on render-times;
    When the load on a server increases, the response times of various parts of a web application may increase as well. These increased response times can have implications on the render time of the web application. For example: a web application heavy with Ajax requests is put under load, the server response times increase, this may result in all Ajax-requests becoming slower, therefore making the website extremely unfriendly to the end-user. When you just run a JMeter script, this will hardly be noticed, and if you do notice it, you cannot express the impact it will have on the user. You can merely speculate about it.
  • Impact of server-side load on functional behavior;
    Given that the server is experiencing increased load and therefore the business-logic of the application is working hard to handle all requests effectively, it can be safe to say the underlying database may also be stressed and therefore responding slower than expected. Slower response times of both application-logic and database requests can result in buggy behaviour of the application. For example incomplete data returned, or worse, a time-out on data or application-logic. How does the application deal with that? How are these errors reported to the end-user? Will the application still function normally within the browser when certain aspects of the application platform are malfunctioning? The best answer to this is by testing the functionality thoroughly while the application is under load. An easy way to test this repeatedly and consistently is by automating these functional tests, for example using (part of) the automated regression test while the servers are under increased load.
  • Advantage of screenshots of fully rendered pages and possible errors with the application under load;
    As a result of the two points mentioned above, it may be extremely useful for both developers/system engineers and your customer to see errors on the pages affected by the increased server load, such as stylesheets not loading or not loading properly, JavaScript not loading, images missing etc. Screenshots (or screencaptures in movie format) will help make clear to the customer what the problem is and more importantly how big the impact on the end-user will be.

I have listed 3 reasons why combining JMeter and WebDriver can be a good idea. I’d love to hear your suggestions of more reasons to want to combine the two.

In a follow-up post I will go into more detail on ways to achieve an effective combination of JMeter and WebDriver running along side each other, well timed and generating consistent logging and results.

JMeter and remote servers – a tutorial

In my previous post I discussed why you might want to run your own servers for load & performance testing. In this article I will elaborate a bit on how to setup your machines. The Apache Jmeter pages of course have an explanation on how to setup remote tests. What I have heard from colleagues and what I experienced myself, is that that explanation is not always as clear and concise as may be preferred. This is my attempt at giving a readable explanation on how to setup JMeter with remote instances.

Some basic terminology

I will use the terms Local and Remote to identify the different sides of the configuration needed to get things working.

  • Local is to be read as your workstation, e.g. the machine you use to build your JMeter scripts.
  • Remote should be considered any machine that will be running a (headless) JMeter instance and will help generate load on an object.

jmeter Client - Service image

How to setup JMeter locally to work with remote machines

Continue reading

JMeter – private remote servers or a service?

Over the past year I have spent a lot of time in close collaboration with Apache JMeter. I used JMeter as a standalone platform, in collaboration with several tools out there and with my own servers. Over the next few posts I will elaborate a bit more on what I have learned from working with tools like BlazeMeter, Loadosophia as well as working with your own remote servers and Jmeter. On top of that I will share some things I consider good practices to use when working with JMeter and building elaborate test scenarios in it.

JMeter and remote servers

Apache JMeterJMeter is great as a stand-alone load generator, however the moment you need to generate a substantial load or a distributed load, you will require the use of more servers. There are several possibilities to add external servers to JMeter. You can use a loadgeneration platform such as BlazeMeter and Flood.io or you can choose to use your own machines (Disclaimer: I have only worked with Flood and BlazeMeter as services, so I only mention these two. There may very well be other services out there I am not (yet) aware of).

Why use your own remote servers?

Virtual Private ServerAlthough there are several commercial options to run JMeter tests from a cloud-platform, it might be useful to have your own servers running. Your own machines, for example Virtual Private Servers, will give you at least the following advantages, please bear in mind these are the main reasons for me to run private servers. It may very well be that you have different reasons for doing the same (I’d be happy to hear them btw!):

  • Availability within firewalls: Not all applications are available outside of firewalls. For quite a lot of companies it is not an option to allow external addresses, especially cloud services, within their firewalled environment. When this is the case in your organisation it can be very useful to have your own load generator hosted inside the firewall.
  • Geographical location: not each and every country has platforms available for load generation. Although platforms like Blazemeter have a lot of different locations available, it may very well be that your particular country is not available through a serviceprovider. However getting your own, hosted, machines within a country is generally not too difficult to do. Keep in mind though whenever you want to use something like a Virtual Private Server to verify with the hosting company whether they allow load generation from within their network!
  • Control: a fairly straightforward reason is in order for you to have full, unlimited control over the load generating servers. If you have your own (virtual) servers running you can easily adjust settings, properties and other things in order to make it fully match your (customers) needs.

Why use a service?

BlazeMeterflood.io

Even though there are quite some reasons for using your own machines, the advantages of using a service like BlazeMeter or Flood.io  are plenty, I will just highlight the few that for me have tipped the scales several times in favor of a service.

  • Maintenance: Setting up your own servers is time consuming and therefore not cheap. These servers need maintenance even though you quite likely do not constantly need them. A Service is just there. You buy a subscription, be it with a time constraint (monthly) or a load constraint (max amount of concurrent users) or something like that, but the service is simply at your fingertips. The moment you need it, you have it.
  • Multiple locations: the JMeter based services generally make use of the Amazon AWS cloud, thus giving you, the user, an immense amount of servers at your disposal. With this comes a huge worldwide distribution, which can be very advantageous for big world-wide used applications.
  • Support: both mentioned platforms have a very solid support base for their customers. If you have questions, if your tests seem to not run properly or you simply have issues getting something done, there generally is a good support platform available. You can use forums or the actual helpdesk, but either way, there is a good, solid, commercial support-base to help resolve your issues.

In the end, I choose per assignment what best fits my needs. Sometimes I use the VPS, sometimes I use a service. It is important you at least think about what best suits your needs for the test you are about to build and execute.

Reusable components in JMeter

Why do I need reusable components in a load script?

Ok, so generally speaking a load script may not really need reusable components you think. However there are plenty of reasons to actually use them and use them effectively, efficiently and heavily.

First of all, not all Jmeter load scripts are for a one off effort. There are quite some times that, if not entire scripts, then at least part of a script could be extremely useful to reuse. Think of a nice application, such as a webshop, you may be trying to stress out with Jmeter. While building it I hope the developers were thinking of reuse, so why wouldn’t you while building a script to stress that shop out?

Secondly, when building load scenarios, you may want to reuse a search function, a registration function, a login function, etc. So why not make a bunch of reusable modules or JMX scripts that help you build scenarios by just clicking existing things together rapidly rather than reinvent the wheel over and over for each and every scenario.

I am not even going to mention the default reasoning: standard practice of DRY: Don’t Repeat Yourself! (ok, there, I did it anyway)

Does JMeter allow for it?

Well, that one seems a bit obvious, since I am writing about it here of course 🙂 Yes Jmeter has a wonderful way of dealing with reuse:

All you need is a Module Controller

Jmeter 2.13 Module Controller

or an Include Controller (which I will write a brief how-to about in a followup post):

Jmeter 2.13 Include Controller

Jmeter 2.13 Include Controller

These two can basically do a similar thing, you can reuse either a module or reuse, and by reuse I mean include of course) a complete testplan in a test. The actual working and thus their use and usefullness is quite different though.

The Module Controller keeps everything within one testplan, whereas the Include Controller expects you to have several (at least 2) testplans to use and work with.

How do I use them

Module Controller

The Module Controller is a relatively easy one to explain as the first one. I use this method often when building scenarios which are relatively close to each other, for example for a webshop. The example I am going through below is a very simple basic one. Sticking to a basic description, since I mainly wanted to show that it is possible to use reusable components and how you can do it.
Imagine a few scenarios for a webshop:

Scenario 1:

  1. Create account
  2. While logged in browse the webshop
  3. Add items to your shopping cart
  4. Go through Checkout
  5. Log out of webshop

Scenario 2:

  1. Browse to the webshop
  2. Add items to your shopping cart
  3. Go to Checkout
  4. Create account on Checkout
  5. Go through checkout
  6. Log out of webshop

Scenario 3:

  1. Browse to the webshop
  2. Sign in to the webshop with an account with at least 1 purchase
  3. Do something the My Account
  4. Add items to shopping basket and leave them there
  5. Log Off

In these scenarios there are several steps that are repeated: Sign up for an account, log off, add some items to a basket. If you want to make sure you do not need to maintain these steps at different spots in your script, it make perfect sense to make these steps reusable. This is after all what you would do with test automation as well, right? Let’s do just that!

What do you need in Jmeter:

Testplan
 > <DISABLED>Threadgroup
    > Simple Controller with Signup steps
        > Samplers etc needed to run through the signup flow
    > Simple Controller with LogOff steps
</DISABLED>
  > Threadgroup
    > Module Controller - In this controller you can choose which controller you want to use. Select the Signup Controller
    > Simple Controller
        > Samplers etc. needed to execute the following steps

Your JMX could now look something like this:

JmeterModuleController

Include Controller

 

This will be coming along in a followup post at a later time.

JMeter and Oracle webcenter

I have started working on a project with lots of load & performance testing, which I really enjoy setting up and executing. However one of the tasks is L&P on a new implementation of Oracle ADF WebCenter, which turned out to be somewhat of a headache. The tools I use for L&P are typically Apache JMeter, Fiddler2, BadBoy and possibly The Grinder, JCrawler or LoadUI.

For the testing of the Oracle WebCenter I picked JMeter. 

The site under test is an intranet with a sizeable community using it. Especially start of day has a tendency to give some heavy hits on the most dynamic pages of the site, so these pages were my focus area. Unfortunately these pages are all behind a login. This login is generally, since the majority of these users use IE, done via Single Sign On where WebCenter verifies the user based on the Windows credentials and grants access based on that.

Going through the SSO login is not possible with JMeter, first of all since JMeter is not a browser, it merely acts to a certain extend as a browser and secondly JMeter has no access to my Windows username, at least not as far as I know. So I got served a login page. Usually getting past a login page with JMeter is fairly easy, the login page I received however didn’t like my passing a username and password in the post call via HTTP (intranet, so no SSL used obviously, cause the intranet is “safe”, right?).

Oh well, let’s google how to get through to the WebCenter main pages. It’s Oracle so someone is bound to have written something sensible about it! Googling the terms “Oracle ADF Jmeter” gave me some nice hits. Titles such as “Configuring Apache JMeter specifically for Oracle’s ADF 11g” or even better, a post on the Oracle Blogs “New recordings on using JMeter to test ADF applications” including links to a demo video.

Some excerpt from these posts:

Sometime back I blogged about Stress & load testing web applications (even ADF & Apex) using Apache JMeter. That post dealt with the generic setup of recording a web session and then replaying under load via JMeter.

In the demo video a bit of a delusional idea of recording and then playing back the Load test “… is a free tool for essentially recording any HTTP session … and then you can replay it …”  (this is in the beginning seconds of the demo, so no need to watch the entire thing.

Why do I highlight these two things so much?

In order to answer this I want to first of all thank Chris Muir for his efforts both on his personal blog and on the Oracle blogs to show that JMeter can be used with the fuzzy logic that is the login of Oracle ADF. However I strongly disagree with the approach of recording and playing things back, especially with an application containing a lot of  ” fuzzy configuration you must get exactly right otherwise ADF gets confused by the messed up HTTP requests it receives from JMeter” as Chris states himself.

Since there is so much fuzziness, to stick to his terms, going on in ADF, it is a lot more sensible to try an understand what it is ADF is looking for. In order to do this one should not record and playback. When recording and playing something back you rely on the “magic” of software. You are not sure what you’re doing, let alone that you can make any clear assumption on what has been “tested” if anything at all for that matter.

So, how do you go about learning how the ADF platform works without having developer access to the beast? You start dissecting it bit by bit. This is why I love doing load & performance testing, it is like solving a puzzle.

I started off by walking through the login flow in Firefox with FireBug switched on. Firebug integrates with Firefox to put a wealth of web development tools at your fingertips while you browse. You can edit, debug, and monitor CSS, HTML, and JavaScript live in any web page.
Firebug gave me the URL’s I needed, so I started to get an understanding of the calls being made, but it didn’t provide me sufficient input yet for getting through the full login flow. In order to get on with this I needed to go a bit deeper, I needed Fiddler.

Fiddler is a Web Debugging Proxy which logs all HTTP(S) traffic between your computer and the Internet. Fiddler allows you to inspect traffic, set breakpoints, and “fiddle” with incoming or outgoing data. Fiddler includes a powerful event-based scripting subsystem, and can be extended using any .NET language.Fiddler is freeware and can debug traffic from virtually any application that supports a proxy, including Internet Explorer, Google Chrome, Apple Safari, Mozilla Firefox, Opera, and thousands more. You can also debug traffic from popular devices like Windows Phone, iPod/iPad, and others.

Passing all traffic through Fiddler, with a filter on the specific domain I was working on made things even more insightful than they had been with Firebug. I all of a sudden noticed the HTTP headers being passed around in my browser differing from the ones in JMeter, so I adjusted JMeter to pass all the same information. The header manager now contains something like the example shown here. Where one of the most important things to add is the referer.

The fact that I made the rest of the header act fully as if it is a webkit or Mozilla browser makes little to no difference, the main thing to do it ensure you add a referer URL to the HTTP headers. This referer needs to be constantly set dynamically, if you need to figure out how to do it, Nabble.com has an excellent explanation for it, but in short it comes down to this:

Thread Group
+HTTP Header manager (Referer = ${Referer})
+set the initial value of the variable Referer as blank or deal with the
first request differently (for e.g. if you dont want the referer at all) in
wh
+Simple Controller
Request 1
Request 2
Request 3
....
++Regular Expression Extractor (as a child of simple controller so that it
applies to all requests)
Name = Referer (same as the header manager variable)
Choose URL in the radio button ,
Expression=(.*)  Template=$1$ (i.e. we get the whole URL)

Get this part right and the rest of building your load test for Oracle ADF is going to be a lot easier. As stated before, the Oracle ADF stuff is not intuitive, getting the HTTP headers right however will help a lot in getting on the right path to logging in.
Besides the headers some more information is, or at least can be, required to get through the login flow. For our configuration I’ve had to deal with a total of 13 redirects which had to be rebuilt from scratch, during the first batch of redirects all kinds of information is gathered about the user session and the state of the user in a set of variables which are needed to be passed on, so be ready to add some nice RegExp extractors to URL’s here and there.

These redirects should be executed by JMeter as a next step, not by Follow Redirects. So please make sure you have that switched off. If you do not do this you will get in a state where ADF says the user session has expired, which is just ADF’s way of saying it doesn’t know who the current session is because the ADF HTTP state parameters JMeter is sending to the ADF server are not what it expected.

It took some trial and error to get it working, but by building it fully by hand rather than by using record/playback (which is not a good option for any type of reusable automation in my view) I learned what ADF is expecting and I now have a set for load & performance testing which I can leave with my customer without having it stuck to a particular session, base URL or user. It is fully reusable and can be run by anyone who can fill in a username and password and then hit the Save and Run buttons.

The difficulty of course lays in the tail, interpreting the outcomes of the test. But that is inherent to Load and Performance testing!