JMeter and WebDriver – 2 ways to combine them effectively

In my previous post I wrote about why it can be useful to combine running load tests in combination with functional automated tests (checks). In this post I will go in a bit deeper and give some ways to combine the two effectively and efficiently. Apache JMeter plus-2-256 Selenium Logo

2 ways to combine JMeter and WebDriver effectively

The two ways to combine Jmeter and WebDriver I will describe differ quite a bit. Depending on your situation you may want to choose one or the other for your purposes. Keep in mind, these are ideas of how I have used the two effectively together, this is not necessarily the best way for you!

Existing webdriver tests

Quite a lot of organisations already have a fairly solid base of testautomation in place, in various states of success. A relatively easy way to get performance metrics for your application is to add a timer to your existing webdriver automation scripts. For every (relevant and useful) action in the browser, you can add a start and done timer, resulting in the actual wait-time within the browser. This timer can be in an “always on” state within your automation suite. It will generate quite some data, which is not a bad thing. When you keep track of your metrics over the course of a complete cycle, for example throughout a sprint, these metrics will help give you and your team insights into possible performance regression.

Performance regression over several releases – Image linked from devel.kostdoktorn.se

While this will give you valuable information, it does not yet give you hard data of how the application runs when put under stress. In order to do that, you will have to create a separate load suite. The load suite will need to contain functional tests, similar to the ones you are running during regular (automated) regression. This way you are certain you are putting your business logic under stress and thus may slow the application down significantly. If your test environment is representative enough, or if you actually benchmarked it beforehand, you can add the Jmeter script to your automated run at planned intervals to start building up load on the server prior to the start of the full browser automated test run. Otherwise on a representative environment you can run the JMeter scripts to generate load, once the target load is reached, you can start your regular automated functional tests,

Advantages

The main advantage is that you are actively encouraged to reuse your existing automated tests. The tests under load are easily compared to the benchmark you make on a daily basis without load.

Challenges

Unfortunately there is no easyway to get to unified reporting other than to generate a set of CSV files and merge those.

Stand-alone performance measurement

As a consultant I also often end up in an organisation where there is no functional automation to reuse for a performance test. In this case I use a fully JMeter based solution for executing the performancetest, Both load generation as well as page-load times (e.g. browser rendering included) are run from JMeter. jmeter-pluginsIn order to run browser-based tests from JMeter I use the wonderful work of the people over at Jmeter-Plugins. They have created a host of wonderful plugins, but for this particular purpose I am very fond of their WebDriver Set.

The WebDriver plugins in JMeter allow you to write JavaScript testscenarios in Jmeter, which actually use the browser. This combined with the regular load tests from within Jmeter make for a strong performancetest.

Advantages

With this solution you achieve a very easy and quick way to get one unified reporting, from within the toolset showing the performance of your application, both the server response times as well as the browser response times.

Challenges

If you already have a testautomation suite running as well, taking this road implies setting up a second testsuite specifically tailored for performancetesting. The lack of reuse of existing tests makes this quite often the more expensive way of performance testing.

When is automation useful?

Recently I helped write a proposal aimed at investigating whether or not to automate regression testing of a system is worth the investment. Looking at their current time spent in regression testing I started wondering when it is worth automating the hell out of everything and when it is just eating money on being hip to have tests automated.

I guess it all depends on a combination of things, such as time spent in regression testing, effort and investment needed to automate things, knowledge about test automation in house (e.g. does an external need to come in and do it for you?), how often do you need to run through regression on the platform, can all of regression be automated or just a part of it etc.

So, seems like you have to be a bit of a mathematician to see if test automating is worth your while, doesn’t it?

Let’s get realistic then instead.

I truly believe the easiest way to see whether test automation is your piece of cake, there’s one very effective way to do it: set a clear goal on what you want to achieve with test automation and based on that do a Proof of Concept. In other words, try it out. This is assuming you have the knowledge to write the code for test automation in house.

Start off with a little bit of research to figure out which tool will quite likely work best for you, there are plenty of Forums out there, or simply ask some fellow testers out there. Take the tool, or for all I care the demo version of some tool, you hope will work for you and try it out. Automate a few tests, just to see how fast it can be done. Do not try to make it all pretty, perfect and reusable immediately. First focus on what’s important: will it work? Does it help me get to my goal?

If it does, cool, now you can start for real, refactor the PoC-code or simply throw it out and start from scratch.

This of course becomes a slightly different story if you do not have the knowledge in-house to write the code yourself, or if you simply have no clue where to start with test automation.

If this is the case, please do not adjust the approach all that much. It is still the best, in my view, to let someone prove to you that their approach and their choice of tooling indeed is the best for your application and organisation. Keep in mind that at the end of the line, you still need to be able to use and maintain the tool and automated tests yourself, in-house. Having some contractor for that running around constantly makes the test automation effort an extremely tedious and expensive one.

Are we building shelf-ware or a useful test automation tool?

Frustration and astonishment inspired this post. There currently is a big regression testing cycle going on within the organization, over the past 4 months we have worked hard with testers to establish a sizable base of automated tests, however the moment regression started everyone seemed to drop the automation tools and revert back into what they have always done: open excel and check the check-boxes of the scripted tests.

Considering that we have already setup a solid base with a custom fixture enabling the tests, or checks if you will, to do exactly what the tester wants them to do and do what a tester would do manually whilst following the prescribed scripts, and having written out, in FitNesse, a fair share of these prescribed scripts, what is stopping them from using this setup?

Are we automating for the sake of automating?

While working on this, extremely flexible, setup with FitNesse and Selenium WebDriver and White as the drivers I have started wondering more and more why we are automating in this organization. The people responsible for testing do not seem to be picking up on the concept of test automation, they are all stating loudly that it is needed and that it is great that we are doing it, but when regression starts they immediately go back to manual checks. I say manual checks on purpose since the majority of testing here is done fully scripted, most of these scripts do not leave anything to the testers imagination, resulting in these tests being checks rather than tests. Checks we can execute automatically, repeatedly and consistently with tools such as FitNesse.

How do you make testers aware that a lot of the scripted tests should not be done manually?

Let me be clear on this, I am a firm believer in both manual and automated testing. They both have their value and should be used together, automated testing is not here to take away the manual testing, rather it is here to support the testers in their work. Automated testing should be complimentary to manual testing. Thus far in this organization, I have seen manual testing happening and I have seen (and experienced) a lot of effort being put into writing out the automated tests in FitNesse. However there has not been a clear cooperation between the two, despite the people writing the automated tests being the same individuals who also are responsible for executing the manual tests (which they have rewritten into FitNesse in order to build automated tests).

We have tried coaching on the job, we have tried dojos, but alas, I still see a hell of a lot of manual checks happening instead of FitNesse doing these checks for them. What is it that makes people not realize the potential of an automation tool? Thus far I have come up with several possible causes

  • In our test-dojos we mainly focused on how to write tests in FitNesse rather than focusing on what you can achieve with test automation. This has led me to the idea that we rapidly need to organize another workshop or dojo in which the focus should be on what the advantages of automated tests are.
  • Another reason could be that test automation was not initiated by this team, it was put upon this team as a responsibility. The team we are currently creating this fixture for is a typical end-of-the-line-bottom-of-the-testing-chain team, everything they get to test is thrown over a wall and left to them to see if it works appropriately. Most of them do not seem to have consciously chosen to be testers, instead they have accidentally rolled into the software testing field. Some of them have adapted very well to this and clearly show affinity and aptitude for testing, others however would, in my opinion, be better of choosing a different occupation. It is exactly the latter group that needs to be pulling this test automation effort currently going on.
There are more reasons I could go into here, but I believe these two to be the main issues at hand here which can actually be addressed.

So what will make people use automation tools properly?

The moment I can answer this one in a general rule-of-thumb I will sell it to the highest bidder. For within this organization however there doesn’t really seem to be a simple solution just yet. As I have written before, there is not yet one sole ambassador for test automation in this organisation. Even if there is, we will need to cause a shift in the general mindset of the testers. Rather than just walking through their predefined set of instructions in excel, they need to consider for themselves what has already gotten covered in the automated tests, how can I supplement these tests with manual testing?

We will need to find a way to get the testers to step out of their comfort-zone and learn how to utilize tools other than Excel and MS Word. Maybe organizing a testing competition will work, see who can cover the most tests in the shortest time and with the highest accuracy?

I am not a great believer in measuring things in testing, but maybe inventing some nice measurements will help the testers see the light. For example “How often can you test the same flow with different input in a certain timeframe?”.

Did we build shelf-ware or did we add value to the testing chain?

At the moment I often ask myself whether I am building shelf-ware or actually am building a useful automation tool (trying to stay away from terms like framework, since that might only increase the distance between the tool and the testers). Whenever I play around with the FitNesse/WebDriver/White setup we currently have running I see an incredibly versatile test automation tool which can be used to make life a lot easier for those who have to test the software regularly and repeatedly (not just testers, but also developers, product owners etc. can easily use this setup).

It is completely environment agnostic, if needed we can (and have in the past) run the same tests we run in a test environment also in production. It is easy to build new test cases/scripts or scenarios (I seem to have lost track what would be the safe option here to choose, they all have their own subconscious connotations) since it is a wiki. All tests are human readable, if you can read an excel sheet, reading the tests in FitNesse with Slim the way we built it, should be child-play.

Despite all these great advantages, the people that should be using it are not.

Reading all this back makes me consider one more thing; we started off building this setup with these tools based on a request from higher management. The tool selection was done by the managers (or team leads if you will) and not by the team themselves. Did we miss out on the one thing the IT industry has taught us? Did we build something we all want, but not what our customer wants and needs? I hope not, for one thing, I am quite sure this is what they need, an easy to use tool to automate all tedious, repetitive check work.

Question that remains: is this what our customer, or to be more exact, our customers’ end user, the tester, wants?

Test automation ambassador needed

I need to define the role of a test automation ambassador.

What would a true, organization, test automation ambassador look like? Should it be a person with lots of experience, especially in the automation field, or can it also be a rookie, fresh out of college?

I guess it all depends on the organisation, right?

Imagine a sizable technology organization where several IT departments live, they all have their own product owners, their own business owners, their own development teams and thus also for a big part their own codebase.

Where does this all come from? We work with separate teams within the IT departments, these teams are completely separated. They have their own product owners, their own business owners, their own development teams and thus also for a big part their own codebase.

From the test automation side we have a unique position within the organisation, we have a helicopter view. We are separated from all these other departments and have our own thing todo: open up all the pieces of the platform to enable test automation with FitNesse and whatever drivers we need. This position creates a lot of perks, such as the freedom to work independently from any of the teams, we get to work first with new tools and toys since we are seen as a bit of a playground.

It of course also poses a potential problem. We are implementing test automation as externals, we are there to help, not to own. Where do we now place the ownership of test automation, and of the FitNesse part in particular, needs to have a place. In my view test automation ownership should not be laying within any of these separate teams, it should be outside of those.

Why does the ownership need to lay outside of the teams?

For one, the teams are competing for resources, another problem would be that they all have different agendas and deem their own part of work the most important contribution to the IT landscape within the company, in other words, their own piece of the automation pie will be well maintained, the rest will be neglected. Most importantly however, non of these teams have a clear overview of what the entire platform, i.e. all components together, look like from a functional point of view.

The owner of the FitNesse side of test automation needs to have this overview. This same owner however, also needs to clearly have some weight to put in the scales to ensure that all teams

  • use test automation effectively
  • maintain their part of the test suite (FitNesse test cases)
  • maintain and add their part of the custom fixture
  • do not break the overall regression, or smoke test or end-to-end test

Logically the owner of test automation would be within the group responsible for regression testing. This is exactly where the challenge is however. The team responsible for regression (let’s call them the regression team), is understaffed and fairly inexperienced, completely inexperienced when it comes to test  automation and how to successfully roll that out over an entire tech organization. The separate team members all have their own strengths and as a team do a very solid job in manually executing regression or smoke tests, but there does not seem to be one person strong enough to pull the automation effort beyond writing the testcases.

Ideally the ambassador of test automation (in this organization at least), and thus the owner of both the FitNesse fixture and the FitNesse testcases resides outside of all teams that have a use for test automation. This opens up the road to continuous development and maintenance on the automation suite, it will ensure independence of other teams and thus the ambassador will be able to make clear decisions based on what is good for the automation program first and think about the teams second, the individual teams are contributing to both the Fixture and the regression suite anyway, so their needs will be covered within sprint or roadmap.

I fear we will have to apply the polder-model here however, and find a way to make it work. What we have built thus far, both as a custom fixture and in terms of testcases, can already be of huge added value to the organization.

However I still hope we can find the prospect ambassador and coach, shape and train this person to have both the knowledge, skills and mental strength to take the next steps needed to get this organization closer to continuous testing.

Awareness of what test automation can do however is holding people back from using it.

– Edit –

So what skills should this ambassador at least possess ?

  • Affinity with testing and test automation in particular
  • Solid understanding of what can be tested automatically and more importantly what should and should not be tested automatically
  • Capability of explaining  to all levels within the organization what we can achieve with test automation
  • Presence and charisma to not just sell test automation within the organization to sceptics, but convince them and show them the added value of it and make them want to use it
  • Insight in how to maintain test scripts across the teams, how to deal with the inheritence from several teams into regression and how to organize this all into a solid, robust, trusted automated regression set
  • At least a base knowledge of programming in order to help maintain the FitNesse fixture and to be able to help new test engineers get started with the inner workings of the fixture
  • Know and understand how the organization works together and how you can get the several teams to contribute effectively to the test automation effort