JMeter and WebDriver – Why would you want to combine them?

3 Reasons to combine JMeter and WebDriver in performance testing a webapplication


What is JMeter good at?

The Apache Foundation states the following about JMeter:Apache JMeter

The Apache JMeter™ desktop application is open source software, a 100% pure Java application designed to load test functional behavior and measure performance. It was originally designed for testing Web Applications but has since expanded to other test functions.

So in other words, it’s a tool you can use to generate functional load on an application or a platform. This also immediately describes what it is good at: generating load. Yes if implemented well you not only generate dumb-load but also hit the functional application layers with the tool. But the basic function of JMeter is aimed at generating load and measuring the (server) response times during this load.

What does JMeter NOT do?

Despite being capable of generating functional load, JMeter does not render pages nor is it very well equipped to execute embedded JavaScript (it is simply not equipped to do that actually), therefore JMeter will not tell you anything (out of the box) about the render times of pages. Especially not about render times when the server is heavily overloaded by your scripts!

What is WebDriver good at?

SeleniumHQ gives a wonderful description of what Selenium (or nowadays WebDriver) does:
Selenium Logo

Selenium automates browsers. That’s it! What you do with that power is entirely up to you. Primarily, it is for automating web applications for testing purposes, but is certainly not limited to just that.

In short, what Webdriver does is just about anything that happens within your browser. It does render pages, it does execute JavaScript, it retrieves the pages as if an actual human was clicking on a website. So for fully functional automated testing (or checking to stick with the more correct terminology) WebDriver is perfect.

What does WebDriver NOT do?

Well, it is not quite good at generating load. Since WebDriver basically requires a browser (yes, it is possible to run it headless of course) it is very difficult to generate multiple (virtual) users. That would require a bunch of browsers to start up, when talking about 10 users that may seem feasible, however when talking about generating real load (say several 1000’s concurrent users) a bunch of browsers becomes a lot more difficult to arrange.

Why combine them then?

The logical question then indeed is, why would you combine them? Below I have set out 3 clear reasons why combining JMeter and WebDriver scripts can be an excellent idea.

  • Impact of server-side load on render-times;
    When the load on a server increases, the response times of various parts of a web application may increase as well. These increased response times can have implications on the render time of the web application. For example: a web application heavy with Ajax requests is put under load, the server response times increase, this may result in all Ajax-requests becoming slower, therefore making the website extremely unfriendly to the end-user. When you just run a JMeter script, this will hardly be noticed, and if you do notice it, you cannot express the impact it will have on the user. You can merely speculate about it.
  • Impact of server-side load on functional behavior;
    Given that the server is experiencing increased load and therefore the business-logic of the application is working hard to handle all requests effectively, it can be safe to say the underlying database may also be stressed and therefore responding slower than expected. Slower response times of both application-logic and database requests can result in buggy behaviour of the application. For example incomplete data returned, or worse, a time-out on data or application-logic. How does the application deal with that? How are these errors reported to the end-user? Will the application still function normally within the browser when certain aspects of the application platform are malfunctioning? The best answer to this is by testing the functionality thoroughly while the application is under load. An easy way to test this repeatedly and consistently is by automating these functional tests, for example using (part of) the automated regression test while the servers are under increased load.
  • Advantage of screenshots of fully rendered pages and possible errors with the application under load;
    As a result of the two points mentioned above, it may be extremely useful for both developers/system engineers and your customer to see errors on the pages affected by the increased server load, such as stylesheets not loading or not loading properly, JavaScript not loading, images missing etc. Screenshots (or screencaptures in movie format) will help make clear to the customer what the problem is and more importantly how big the impact on the end-user will be.

I have listed 3 reasons why combining JMeter and WebDriver can be a good idea. I’d love to hear your suggestions of more reasons to want to combine the two.

In a follow-up post I will go into more detail on ways to achieve an effective combination of JMeter and WebDriver running along side each other, well timed and generating consistent logging and results.

Thoughts while preparing testingdojo follow up

In the first testingdojo at my current client we familiarized the team with each other and of course with Selenium and Fitnesse for test automation.

As I wrote earlier this was quite a succes, all participants seemed to have really enjoyed the dojo and all learned quite some new tricks on how to use Fitnesse to their own advantages as well as the companies advantages. During this first session we stuck to the basics of what can be done with FitNesse, such as simple testcases, use of variables and some basic reusability like login.

All the fancy stuff we have not even touched upon, so my first thoughts were to cover that during the coming, second, testingdojo.

I am however now doubting that idea. Isn’t it a lot more sensible to explain the basics and have them work, as a team, on what is best-practice for testautomation, how can they implement this best for their organization?

Considering the general lack of knowledge in testautomation within this organization I truly believe this would be the best way to go, however question now is of course, how do you keep a session like that interesting? This could easily turn into a very boring, theoretical discussion rather than a properly interactive dojo.

A thought that occurred to me is that we might start off describing the best practices and then pair them up and make them search for ways to get to these best practices as well as reviewing existing automated testcases and refactor them to adhere to these best practice. In order to do this I would of course need to start off compiling my own list of test automation best practices specially tailored for testautomation within this organization and with FitNesse as a background. So far I have come up with the following list, keeping in mind that my group has limited knowledge of testautomation and will solely work in FitNesse:

  • do you know what we are trying to achieve with testautomation within the organization? Are we automating for the correct reasons?
  • before writing an automated test, describe your objective, does it still make sense to automate this test after reading your objective?
  • choose wisely what to automate and what not, be clear on the reason why you are automating someting
  • keep your tests short, readable and simple in order to keep maintenance low and knowledge tranfer capability high
  • make your tests data-driven, try avoiding hardcoded values. In a fast-moving environment like my customers’ it is key to ensure you are not facing failing tests due to inconsistent or wrong data hardcoded in your tests
  • try thinking in reusable functional pieces, keep an eye open for actions you do more than once and see if it makes sense to a) execute this action more than once and b) if this is indeed the case should we make this function(ality) a reusable scenario?

I am fully confident that I have left out quite a few (even more) important points, however the team itself will need to come up with a list of what they believe is going to be key in making testautomation a success within the entire organization, that is what a testingdojo is about, working together and learning from oneanother.

My first experiences organizing a Testingdojo

A bit of background

While working as a testconsultant at an investment banking firm on a testautomation implementation with FitNesse we were in need of more customer engagement.

The custom fixtures for FitNesse were in place, a base test set in FitNesse had been written in FitNesse, regular runs of these tests were in place and showing beautiful green results. The time had come to get our customer more closely involved again in order for them to add the Acceptance tests and build up a solid suite of automated tests.

Our first attempts at getting people involved led to a few added testcases, but not a feeling within the team that this was going to help them a lot.

The audience

The team we had to work with is a fairly diverse group, ranging from a person who has quite a solid knowledge of coding and working with Selenium to a functional tester with little or no technical knowledge or interest in programming.A group so mixed makes life difficult when trying to engage them all, or so I thought.

When looking for possibilities to get them all together and work on FitNesse tests together I considered several options, such as a workshop kind of setup where we would feed the team the information. However experience has shown me that feeding information is not the best option when you want to actually teach something to a group of people. The most effective way to get them all learning and seeing the benefits we would have to get the whole group together, interacting and playing with the tools. Thus I came up with a testingdojo.

Preparation

In the past I have participated in all kinds of interesting things such as dojo’s, hackathons etc, but never had I participated, let alone organised a testingdojo. All my previous experiences were software development related, with a group of relatively like-minded people.

I started reading up on the base ideas of what a testingdojo can be on the web, starting at testingdojo.org. The base idea behind a testing dojo is similar to that of a codingdojo: you put a group of people together, clarify the roles and expectations, set a challange or goal for the duration of the dojo and set off to work.

Considering the different levels within this group I decided to pair the participants up by their level of interest and base knowledge of FitNesse. In other words, someone without much base-knowledge of the tool was paired with someone with with a fair amount of baseknowledge. This way they were capable of showing oneanother new ideas and insights, especially since the people without the base knowledge also generally did not work with a testautomation tool before. They were a clean slate.

Besides the pairing setup I also prepared a set of testcases they could use as a starting point, I have noticed that within this organization people enjoy getting given a clear direction rather than having an open target. These testcases were also designed to be datadriven and sensible to repeat often, since they all touch the core of the business.

The dojo

The dojo was planned for an entire working day, this in order for everyone to have a chance to get enough experience in and raise their questions.

We started the dojo off with a brief introduction of why we all came together, what the idea behind a testingdojo is and most importantly, what FitNesse is, how it works and what you can do with it. One of the main takeaways that I ensured kept coming back during the entire day was that Testautomation is a tool, it is not a goal. At first this raised some eyebrows and the inevetable questions, but while working with FitNesse during the day the group started to see the point.

Once the introduction of the day was done people set to work with a few rules clearly laid out:

for the duration of this dojo silence is bad. Talk, have conversations, explain to your partner what you are doing, why you are doing it and what you hope to gain out of it.

there are no stupid questions. The only stupid question is the one you do not ask. If the answer you get is not clear enough, seems not to answer your question or does not suffice otherwise for you, keep on asking for clarifications until you get an answer you can accept.

a FitNesse testcase is DONE when it is on the server, in a logical spot within the suites and has 2 consecutive succesful runs.

The goal set for the group was: farmiliarize yourself with FitNesse and evaluate it, determine the added value this tool can have for you and how you can use testautomation. Create testcases that will be useful in your opinion to have running in a testautomation suite and will make your life as a tester easier.

The outcome

Beforehand I expected we would need to do a lot of coaching and helping to get started as facilitators of the dojo. My expectations were proven wrong thankfully! The pairs hit it off very good, the people with no or hardly any experience were at the keyboards playing with Fitnesse and writing tests while the other halves of the pairs, as observers and recorders, were indeed busy asking questions, coming up with ideas and suggestions, explaining what is and what is not useful to do with testautomation.

The people with little or no experience, by the end of the day, were enjoying themselves a lot writing testcases and seeing them run succesfully. The people with quite some experience also seemed to quite enjoy the feeling of having tought someone else a new skill, besides having gotten new insights into what other testers think is an important testcase.

Overall we all had the feeling this has been a great excercise both in learning some new skills, seeing more sense in testautomation and bringing the team closer together and collaborate more as a team, rather than as a group of individuals.

Next steps

We have already planned a second testingdojo session in which we will continue this line of working and thinking, this second dojo will be focussed more on the specifics you can do with FitNesse, such as working with suites, scenarios and make a test fully and truly datadriven, how to debug your test and stabilize it etc.