JMeter and WebDriver – 2 ways to combine them effectively

In my previous post I wrote about why it can be useful to combine running load tests in combination with functional automated tests (checks). In this post I will go in a bit deeper and give some ways to combine the two effectively and efficiently. Apache JMeter plus-2-256 Selenium Logo

2 ways to combine JMeter and WebDriver effectively

The two ways to combine Jmeter and WebDriver I will describe differ quite a bit. Depending on your situation you may want to choose one or the other for your purposes. Keep in mind, these are ideas of how I have used the two effectively together, this is not necessarily the best way for you!

Existing webdriver tests

Quite a lot of organisations already have a fairly solid base of testautomation in place, in various states of success. A relatively easy way to get performance metrics for your application is to add a timer to your existing webdriver automation scripts. For every (relevant and useful) action in the browser, you can add a start and done timer, resulting in the actual wait-time within the browser. This timer can be in an “always on” state within your automation suite. It will generate quite some data, which is not a bad thing. When you keep track of your metrics over the course of a complete cycle, for example throughout a sprint, these metrics will help give you and your team insights into possible performance regression.

Performance regression over several releases – Image linked from

While this will give you valuable information, it does not yet give you hard data of how the application runs when put under stress. In order to do that, you will have to create a separate load suite. The load suite will need to contain functional tests, similar to the ones you are running during regular (automated) regression. This way you are certain you are putting your business logic under stress and thus may slow the application down significantly. If your test environment is representative enough, or if you actually benchmarked it beforehand, you can add the Jmeter script to your automated run at planned intervals to start building up load on the server prior to the start of the full browser automated test run. Otherwise on a representative environment you can run the JMeter scripts to generate load, once the target load is reached, you can start your regular automated functional tests,


The main advantage is that you are actively encouraged to reuse your existing automated tests. The tests under load are easily compared to the benchmark you make on a daily basis without load.


Unfortunately there is no easyway to get to unified reporting other than to generate a set of CSV files and merge those.

Stand-alone performance measurement

As a consultant I also often end up in an organisation where there is no functional automation to reuse for a performance test. In this case I use a fully JMeter based solution for executing the performancetest, Both load generation as well as page-load times (e.g. browser rendering included) are run from JMeter. jmeter-pluginsIn order to run browser-based tests from JMeter I use the wonderful work of the people over at Jmeter-Plugins. They have created a host of wonderful plugins, but for this particular purpose I am very fond of their WebDriver Set.

The WebDriver plugins in JMeter allow you to write JavaScript testscenarios in Jmeter, which actually use the browser. This combined with the regular load tests from within Jmeter make for a strong performancetest.


With this solution you achieve a very easy and quick way to get one unified reporting, from within the toolset showing the performance of your application, both the server response times as well as the browser response times.


If you already have a testautomation suite running as well, taking this road implies setting up a second testsuite specifically tailored for performancetesting. The lack of reuse of existing tests makes this quite often the more expensive way of performance testing.

Heb je die testautomationtool wel nodig?

3 tips om tot een goede toolselectie te komen

Als testconsultant kom ik vaak bij klanten over de vloer die een testautomatiseringstool willen aanschaffen. Ze stellen mij dan regelmatig de vraag “welke tool moeten we kopen of gebruiken?”.  Ik antwoord dan meestal met een tegenvraag: Waarom wil je een testautomatiseringstool hebben? De reactie die ik hierop krijg is redelijk voorspelbaar: grote ogen, gefronsde wenkbrauwen,  verbijstering op de gezichten dat een consultant zo’n onnozele vraag stelt en vervolgens de (steeds falende) poging om een antwoord te geven. GO BACKAntwoorden die ik krijg gaan vaak in de richting van:

  • “Ik heb er goede dingen over gehoord of gelezen”
  • “Een kennis van me werkt er ook mee”.

Of het nog gevaarlijker:

  • “Bij mijn vorige werkgever gebruikten we ook een testautomatiseringstool”.

Het mooiste antwoord wat ik ook nog wel eens krijg is:

  • “We hebben een demo gehad van een tool en dat zag er heel veelbelovend uit”.

The first principle is that you must not fool yourself — and you are the easiest person to fool.

– Richard Feynman

Deze antwoorden zijn duidelijke indicatoren dat het echte probleem niet duidelijk is en daarmee is ook het doel wat de klant hoopt te bereiken met de testautomatiseringtool niet duidelijk.

Wat is het probleem dat opgelost moet worden?

Waarom stel ik nou juist die vraag? Er zal toch wel een goede reden zijn waarom die klant vraagt om een testautomatiseringstool?

In veel gevallen is de reden van het zoeken naar testautomatiseringstools niet de échte reden, de diepere reden. Door te vragen waaróm men op zoek is naar testautomatiseringstools ga je op zoek naar welk probleem er is. Ik wil weten wat de aanleiding is voor het willen implementeren van een tool en vooral wat de klant hoopt ermee te bereiken.

Ik wil mijn gesprekspartner uitlokken (of uitnodigen) alles boven water te halen.

In veel gevallen is een testautomationtool in eerste instantie een lapmiddel. Het is symptoombestrijding omdat het eigenlijke probleem

  • te groot is
  • buiten jou invloedsfeer ligt
  • niet op korte termijn opgelost kan worden.

Symptoombestrijding geeft geen oplossing voor het echte probleem. Als bewuste symptoombestrijding het hoogst-haalbare is op dit moment, dan is het wellicht een goed idee om met een tool aan de slag te gaan. Vaak echter lijkt een testautomatiseringstool een oplossing te bieden voor iets wat niet de basis van het probleem is en dus wordt het echte probleem niet weggenomen, laat staan opgelost.

Er zijn een aantal dingen waar je naar moet kijken als je een testautomatiseringstool gaat selecteren.

 Wat moet je weten voor je een testautomatiseringstool selecteert?

  1. Definieer de basis van het probleem dat je poogt op te lossen met de testautomatiseringstool
  2. Stel jezelf de vraag “gaat een tool dit probleem daadwerkelijk oplossen?
  3. Hebben we de kennis, kunde en financiële middelen om een en ander ook door te zetten en het probleem écht op te lossen?

 Het is mogelijk om op tal van manieren te falen… terwijl je slechts op één manier kunt slagen.

– Aristoteles

JMeter and WebDriver – Why would you want to combine them?

3 Reasons to combine JMeter and WebDriver in performance testing a webapplication

What is JMeter good at?

The Apache Foundation states the following about JMeter:Apache JMeter

The Apache JMeter™ desktop application is open source software, a 100% pure Java application designed to load test functional behavior and measure performance. It was originally designed for testing Web Applications but has since expanded to other test functions.

So in other words, it’s a tool you can use to generate functional load on an application or a platform. This also immediately describes what it is good at: generating load. Yes if implemented well you not only generate dumb-load but also hit the functional application layers with the tool. But the basic function of JMeter is aimed at generating load and measuring the (server) response times during this load.

What does JMeter NOT do?

Despite being capable of generating functional load, JMeter does not render pages nor is it very well equipped to execute embedded JavaScript (it is simply not equipped to do that actually), therefore JMeter will not tell you anything (out of the box) about the render times of pages. Especially not about render times when the server is heavily overloaded by your scripts!

What is WebDriver good at?

SeleniumHQ gives a wonderful description of what Selenium (or nowadays WebDriver) does:
Selenium Logo

Selenium automates browsers. That’s it! What you do with that power is entirely up to you. Primarily, it is for automating web applications for testing purposes, but is certainly not limited to just that.

In short, what Webdriver does is just about anything that happens within your browser. It does render pages, it does execute JavaScript, it retrieves the pages as if an actual human was clicking on a website. So for fully functional automated testing (or checking to stick with the more correct terminology) WebDriver is perfect.

What does WebDriver NOT do?

Well, it is not quite good at generating load. Since WebDriver basically requires a browser (yes, it is possible to run it headless of course) it is very difficult to generate multiple (virtual) users. That would require a bunch of browsers to start up, when talking about 10 users that may seem feasible, however when talking about generating real load (say several 1000’s concurrent users) a bunch of browsers becomes a lot more difficult to arrange.

Why combine them then?

The logical question then indeed is, why would you combine them? Below I have set out 3 clear reasons why combining JMeter and WebDriver scripts can be an excellent idea.

  • Impact of server-side load on render-times;
    When the load on a server increases, the response times of various parts of a web application may increase as well. These increased response times can have implications on the render time of the web application. For example: a web application heavy with Ajax requests is put under load, the server response times increase, this may result in all Ajax-requests becoming slower, therefore making the website extremely unfriendly to the end-user. When you just run a JMeter script, this will hardly be noticed, and if you do notice it, you cannot express the impact it will have on the user. You can merely speculate about it.
  • Impact of server-side load on functional behavior;
    Given that the server is experiencing increased load and therefore the business-logic of the application is working hard to handle all requests effectively, it can be safe to say the underlying database may also be stressed and therefore responding slower than expected. Slower response times of both application-logic and database requests can result in buggy behaviour of the application. For example incomplete data returned, or worse, a time-out on data or application-logic. How does the application deal with that? How are these errors reported to the end-user? Will the application still function normally within the browser when certain aspects of the application platform are malfunctioning? The best answer to this is by testing the functionality thoroughly while the application is under load. An easy way to test this repeatedly and consistently is by automating these functional tests, for example using (part of) the automated regression test while the servers are under increased load.
  • Advantage of screenshots of fully rendered pages and possible errors with the application under load;
    As a result of the two points mentioned above, it may be extremely useful for both developers/system engineers and your customer to see errors on the pages affected by the increased server load, such as stylesheets not loading or not loading properly, JavaScript not loading, images missing etc. Screenshots (or screencaptures in movie format) will help make clear to the customer what the problem is and more importantly how big the impact on the end-user will be.

I have listed 3 reasons why combining JMeter and WebDriver can be a good idea. I’d love to hear your suggestions of more reasons to want to combine the two.

In a follow-up post I will go into more detail on ways to achieve an effective combination of JMeter and WebDriver running along side each other, well timed and generating consistent logging and results.

How did teaching test automation work out?

A while ago I wrote a post about a set of workshops I was asked to setup for functional testers, test coordinators and test managers to get the familiar and acquainted with testautomation and performance testing. I pre-selected a set of tools which I wanted to go through with the participants. The slides I used for this can be found on SlideShare:

Apologies for the slides being in Dutch. Possibly I will come up with the EN version as well, however the training was in Dutch.

This part of the evening already created a lot of discussion and questions. One of the nicest questions was the obvious one: “is coding really needed when working on test automation? I thought test automation tools and software were advanced enough nowadays to no longer require code to work well?” This questions fairly obvious, considering the tools I selected for the training, Sikuli and AutoIt SciTE. Both tools require a lot of coding in order for them to be at all usable.

The hands-on experiences

After we had had the theoretical bits we moved on to the hands on bit. As visible on the last slide there were a few (simple) assignments prepared for the participants. The first one was executing a calculation with the Microsoft built-in calculator. Fairly straight forward and rudimentary I thought. A few had prior coding knowledge so they went to do this exercise with AutoIt rather than with Sikuli, the majority of the group however attempted to execute this task with Sikuli.


Since Sikuli is mainly image based almost everyone started by actually clicking on the START button and going through the Windows Start Menu. After a while I thought it worth while to show the group how to just launch an application from within Sikuli through the RUN command of Windows. This of course immediately raised the question why I would prefer to do that over manipulating the mouse (which is answered simply by explaining the concept of cross-platform (or at least cross windows version) testing with test automation, the <WIN>+R key-combo has been in existence since Windows 95 if I recall correctly and thus this is backwards compatible and since you can still use this within Windows 8.1 it is also forwards compatible).

Sikuli turned out to be a hit with the participants. They barely noticed they were actually writing code and at some point I saw a fairly experienced test-manager explain some basic things of Python coding to at least two other test-managers. Non of these had prior coding experiences, not even the one explaining things.


AutoIt, with full access and use of the Windows API however was a bit more of a stretch. Turns out the Basic based language for AutoIt is for non-coding testers a lot more difficult to understand than Python. The first assignment, manipulating the Windows Calculator, was for most still doable, although it took a lot more explaining and showing than Sikuli.

The second assignment, calculating the square root of a number in Excel proved for most really difficult. I had hoped they would see the use of the Windows API and thus also come up with using the Office API in doing this, but apparently I overestimated the ease of use of AutoIt for them.

My takeaways

Next time I do an evening like this I now know to introduce AutoIt separately, after people have gained some experience with Sikuli. Guiding them a bit more with the AutoIt things, instead of letting them go.
Overall the test automation evening was really great and I do believe everyone had great fun and actually got a bit of an idea of what it is that attracts me to testautomation.

2013 in review

The stats helper monkeys prepared a 2013 annual report for this blog.

Here’s an excerpt:

The concert hall at the Sydney Opera House holds 2,700 people. This blog was viewed about 10,000 times in 2013. If it were a concert at Sydney Opera House, it would take about 4 sold-out performances for that many people to see it.

Click here to see the complete report.

Teaching test automation & performance testing

I was asked to share some knowledge on test automation and load and performance testing by Polteq. The company is running a “Special Development Program” in which employees from all levels of the company, ranging from junior engineers to senior consultants, get the opportunity to follow trainings and courses in a range of things. Varying from social skills to hard technical skills. It is the latter I have been asked to help provide some training for, which I do gladly.

warning-mass-confusion-aheadThere is however quite a challenge, as said, the audience ranges from junior to quite senior, but also from technically strong to technically challenged (my apologies to all the colleagues I am insulting with this statement, but I know you can forgive me:) ). So how do I go about preparing two trainings, one about test automation and one about load and performance testing, for such a diverse group? The duration of these sessions is also quite limited, making it even more difficult to come up with something sane to do in these evenings.JMeter

Oh! To make my life easier, I have been telling everyone, during a bunch of company meetings and updates over the past month, I do not believe sitting back and listening to someone broadcasting the information will help in learning something to do with technical skills.

Sikuli ScriptThe trainings need to be interactive, but also guided and somewhat personalized. Thus far I have come up with the idea of preparing a bunch of USB sticks with a set of portable applications I use regularly to automate stuff with. When in this context I say automate, I really mean hack something together which does the job and at the end of the project (or my involvement with the project) gets thrown away. Do I indeed want to teach the habit of writing throw-away-code?autoit-logo

On top of that, do I want to teach some “technical” or basic programing skills based on examples with tools, which should in their own right not be used to automate these things? Actually, I believe I do! My goal for these evenings will be to get this group excited to use and abuse tools to their own advantage! The tools I already chose, now I need to figure out some interesting, useful and enjoyable targets for these people to hack their way around. Tips anyone?

Test automation – Finding your starting point

The other day I had a job coaching of a colleague who, for his assignment, needs to start setting up test automation within an agile environment. He is very interested in test automation and the possibilities it gives you, however his knowledge on the subject, as well as on coding, is very limited.

In preparation for this first coaching session with him I was pondering where to start with him and I ended up starting with the, to me, obvious points :

  • just start with a first step
  • KISS
  • DRY

Just start with the first step

My colleague was struggling on how to start and more importantly, where to start with test automation. This is an issue I have faced myself often in the past as well. You have this huge task of automating a system laying ahead of you, there is so much to do and so little time. Where do you start?

Over the years I have come up with a simple strategy for that: just start with the basics and move on from there.

In other words, start with the basic functionality you will need for any test you are about to execute. In most systems that is setting up a simple sequence for validating the home- or landingpage, going to a login screen, validating that loads properly and then logging in.

Login screen


Once you have this start it often becomes easier and more logical to move ahead and start automating other functions and functionalities. In this particular case once logged in there is a landing page which consists of a dashboard, so we scripted the verification of all required items on the dashboard. Once this was done my colleague again was questioning, what do now?

Again, we kept it simple and straightforward, I asked him to name another functionality he often uses and which help add value to the test immediately and make life easier in testing in general. He came up with logging off.

Over the course of this hands-on coaching session we ended up writing several scripts, which when put together in a scenario ended up being a very simple and fast sanity check on the application under test, which can immediately be reused in Continuous Integration and on every new code drop they will be doing on any environment.

Keep it simple and don’t repeat yourself.

Once we got through the initial start we went on to two other very important things you need to keep in mind when automating (or writing any code for that matter): KISS and DRY.


Keep It Simple

Keep It Simple

KISS is well know to mean “Keep It Simple, Stupid” the main thing I generally take from this is the “Keep It Simple” part. For test automation it is important to keep things simple, you need to be able to maintain the suite, others need to be able to understand what you’re doing and why you did things in a certain way. Therefore, I rigorously stick to keeping things as simple as possible considering circumstances. I generally try to follow the KISS rule in all aspects of test automation: naming, coding, hierarchy, refactoring, running etc. Some of the ways this is visible is: I try to keep scripts short and concise, I try to ensure names of scripts clearly state what the script (supposedly) does, I try to keep variables I use clearly names (e.g. I try to stay away from statements like i = 1 , instead I give i a meaningful name).


DRY, or Don’t Repeat Yourself, is a well known practice in software development and should be used within test automation as well. I apply DRY both in the scripts and in my daily work. When starting with automation on any project some of the first things I generally put on my list of things to automate sooner rather than later are all functions and functionalities I have to use more than rarely. In other words, starting an application and logging in I generally automate as quickly as possible, even on a manual testing project!

Don't Repeat Yourself

Don’t Repeat Yourself

One of the reasons I am a strong advocate of test automation is that I am lazy. Why would I want to start an application and login to it manually and thus have to execute a whole bunch of things myself, several times (sometimes tens or hundreds of times a day) when I can automate that same flow in a few minutes and not have to deal with it anymore? Quite often starting and logging in of an application, when automated, is faster than when you do it manually as well, so it not only saves me from repetitive work, it also speeds up my work!

In other words, the DRY principle can also work outside of the testautomation code, in your daily work. If a task is repetitive, chances are you can automate it!