Jmeter Tips & Tricks – Tip 7

Tip 7 – adjusting the JVM HEAP size, why and how to

When running JMeter tests, it may happen that the test is not successful and JMeter crashes with an OutOfMemoryError. This can happen due to all kinds of things, most of the time however you are trying to generate too many threads with an insufficient amount of memory allocated to your Java Virtual Machine. More information on what the HEAP is, how the JVM deals with the HEAP can be found here.

The HEAP size is defined on startup of your java application, in our case when we start JMeter. In other words, adjusting the HEAP means we will need to adjust the startup script. The startup script can be found in your JMeter “bin” directory. In my case, on a linux environment, it can be found here:


When opening the “jmeter” file in your favorite editor, scroll down until you hit a line stating the following (on Windows based systems open “jmeter.bat”):

# This is the base heap size -- you may increase or decrease it to fit your
# system's memory availability:
HEAP="-Xms512m -Xmx512m"

On Windows systems the line looks slightly different, in the jmeter.bat, due to the nature of BATCH files and DOS working differently from Linux systems

set HEAP=-Xms512m -Xmx512m

When you adjust this line to read something like this:

# This is the base heap size -- you may increase or decrease it to fit your
# system's memory availability:
HEAP="-Xms1024m -Xmx4096m"

the HEAP size will by default be larger, meaning you do not have to think about it anymore when starting JMeter. This does pose one possible challenge, it slows down the startup of JMeter somewhat (at least on most machines I have done it). This seems to be because the JVM will first allocate all the memory you defined in the Xms param, ensuring the JVM is at the appropriate size instantly.

So, we changed the values of Xms and Xmx, what do these two mean?

-Xms ==> minimum heap size. This is the value JAVA will attempt to claim in order to run JMeter. This does NOT imply that it immediately gets this fully allocated by the OS. The OS typically doesn't give an application all blocks it asks for until really needed.
-Xmx ==> maximum heap size. This is the value JAVA will use as a maximum size of the JVM. When JMeter passes this level, an OutOfMemoryError will occur once more.

An alternative, more flexible setup to change the heapsize is to pass the JVM arguments on the commandline when starting JMeter. This of course assumes you are comfortable in commandline and start JMeter from commandline.

[user@machine ]$ JVM_ARGS="-Xms1024m -Xmx2048m" jmeter

Be Aware!

Please always make sure you leave enough memory for your OS and other applications to run on your machine when allocating more HEAP space to a JVM. Not leaving enough memory for your OS to keep running will result in your machine simply being non-responsive and thus your test and testresults being rendered useless.


Jmeter Tips & Tricks – Tip 6

Tip 6 – Effective use of Transaction Controllers

Whether you are trying to script a website, a webservice or something else, quite often you find that one functional action results in several requests to a server.

Take for example an average login-request seen from a functional point of view:

LoginScreenA user types in username & password and hits the Login button.
If the username/password combination was valid the user is than shown a dashboard.

This seems straight forward enough from a functional point of view. What actually happens however is a bit more than just this one request:

  • Username/password are sent to an authentication service
  • Authentication service sends an authenticated message back (assuming you are indeed authenticated by the application).
  • The authentication message contains a redirect away from the authentication service to the dashboard a user expects. This redirect contains information such as the AuthToken that was just recieved from the service.
  • Subsequent requests for the Dashboard-contents are sent.

In a simple setup, this may require only 1 request in Jmeter, just the login. Since Jmeter can take care of any redirecting itself.

Making Jmeter responsible for blindly following all redirects however, does not always give you the performance metrics you may want or need. When you need to know exactly which part of the login sequence is giving your users some form of grief, you may want to build these requests completely yourself in Jmeter. That will give you the possibility to see if there is high latency, big loads etc. on a per-request basis.

You may end up with something like this: jmeter-4requests

The 4 requests you see in this example are all separate requests used to log a user into the system and get the base information for the first screen of the application.

This sequence could have been captured in 1 request, Login, since the other requests are handled by HTTP/302 responses.

By building the requests yourself and clustering them together in a Logic Controller you now can see and measure how long the entire login request, step by step, takes. The Logic Controller I have used here is a Transaction Controller, this controller can add all child-transactions into one single response time for you by setting the appropriate flag. TransactionController
When set, “Generate parent sample” will ensure that your statistics have a single statistic for the complete set of requests, yet you can easily gain information on the separate requests while running a loadtest by unchecking this flag and looking into each step by itself as well.

This gives you the possiblity to find out which of the steps slows the entire transaction of logging in to your application (or whichever functional step you define in a Transaction Controller of course) .

Heb je die testautomationtool wel nodig?

3 tips om tot een goede toolselectie te komen

Als testconsultant kom ik vaak bij klanten over de vloer die een testautomatiseringstool willen aanschaffen. Ze stellen mij dan regelmatig de vraag “welke tool moeten we kopen of gebruiken?”.  Ik antwoord dan meestal met een tegenvraag: Waarom wil je een testautomatiseringstool hebben? De reactie die ik hierop krijg is redelijk voorspelbaar: grote ogen, gefronsde wenkbrauwen,  verbijstering op de gezichten dat een consultant zo’n onnozele vraag stelt en vervolgens de (steeds falende) poging om een antwoord te geven. GO BACKAntwoorden die ik krijg gaan vaak in de richting van:

  • “Ik heb er goede dingen over gehoord of gelezen”
  • “Een kennis van me werkt er ook mee”.

Of het nog gevaarlijker:

  • “Bij mijn vorige werkgever gebruikten we ook een testautomatiseringstool”.

Het mooiste antwoord wat ik ook nog wel eens krijg is:

  • “We hebben een demo gehad van een tool en dat zag er heel veelbelovend uit”.

The first principle is that you must not fool yourself — and you are the easiest person to fool.

– Richard Feynman

Deze antwoorden zijn duidelijke indicatoren dat het echte probleem niet duidelijk is en daarmee is ook het doel wat de klant hoopt te bereiken met de testautomatiseringtool niet duidelijk.

Wat is het probleem dat opgelost moet worden?

Waarom stel ik nou juist die vraag? Er zal toch wel een goede reden zijn waarom die klant vraagt om een testautomatiseringstool?

In veel gevallen is de reden van het zoeken naar testautomatiseringstools niet de échte reden, de diepere reden. Door te vragen waaróm men op zoek is naar testautomatiseringstools ga je op zoek naar welk probleem er is. Ik wil weten wat de aanleiding is voor het willen implementeren van een tool en vooral wat de klant hoopt ermee te bereiken.

Ik wil mijn gesprekspartner uitlokken (of uitnodigen) alles boven water te halen.

In veel gevallen is een testautomationtool in eerste instantie een lapmiddel. Het is symptoombestrijding omdat het eigenlijke probleem

  • te groot is
  • buiten jou invloedsfeer ligt
  • niet op korte termijn opgelost kan worden.

Symptoombestrijding geeft geen oplossing voor het echte probleem. Als bewuste symptoombestrijding het hoogst-haalbare is op dit moment, dan is het wellicht een goed idee om met een tool aan de slag te gaan. Vaak echter lijkt een testautomatiseringstool een oplossing te bieden voor iets wat niet de basis van het probleem is en dus wordt het echte probleem niet weggenomen, laat staan opgelost.

Er zijn een aantal dingen waar je naar moet kijken als je een testautomatiseringstool gaat selecteren.

 Wat moet je weten voor je een testautomatiseringstool selecteert?

  1. Definieer de basis van het probleem dat je poogt op te lossen met de testautomatiseringstool
  2. Stel jezelf de vraag “gaat een tool dit probleem daadwerkelijk oplossen?
  3. Hebben we de kennis, kunde en financiële middelen om een en ander ook door te zetten en het probleem écht op te lossen?

 Het is mogelijk om op tal van manieren te falen… terwijl je slechts op één manier kunt slagen.

– Aristoteles

JMeter and WebDriver – Why would you want to combine them?

3 Reasons to combine JMeter and WebDriver in performance testing a webapplication

What is JMeter good at?

The Apache Foundation states the following about JMeter:Apache JMeter

The Apache JMeter™ desktop application is open source software, a 100% pure Java application designed to load test functional behavior and measure performance. It was originally designed for testing Web Applications but has since expanded to other test functions.

So in other words, it’s a tool you can use to generate functional load on an application or a platform. This also immediately describes what it is good at: generating load. Yes if implemented well you not only generate dumb-load but also hit the functional application layers with the tool. But the basic function of JMeter is aimed at generating load and measuring the (server) response times during this load.

What does JMeter NOT do?

Despite being capable of generating functional load, JMeter does not render pages nor is it very well equipped to execute embedded JavaScript (it is simply not equipped to do that actually), therefore JMeter will not tell you anything (out of the box) about the render times of pages. Especially not about render times when the server is heavily overloaded by your scripts!

What is WebDriver good at?

SeleniumHQ gives a wonderful description of what Selenium (or nowadays WebDriver) does:
Selenium Logo

Selenium automates browsers. That’s it! What you do with that power is entirely up to you. Primarily, it is for automating web applications for testing purposes, but is certainly not limited to just that.

In short, what Webdriver does is just about anything that happens within your browser. It does render pages, it does execute JavaScript, it retrieves the pages as if an actual human was clicking on a website. So for fully functional automated testing (or checking to stick with the more correct terminology) WebDriver is perfect.

What does WebDriver NOT do?

Well, it is not quite good at generating load. Since WebDriver basically requires a browser (yes, it is possible to run it headless of course) it is very difficult to generate multiple (virtual) users. That would require a bunch of browsers to start up, when talking about 10 users that may seem feasible, however when talking about generating real load (say several 1000’s concurrent users) a bunch of browsers becomes a lot more difficult to arrange.

Why combine them then?

The logical question then indeed is, why would you combine them? Below I have set out 3 clear reasons why combining JMeter and WebDriver scripts can be an excellent idea.

  • Impact of server-side load on render-times;
    When the load on a server increases, the response times of various parts of a web application may increase as well. These increased response times can have implications on the render time of the web application. For example: a web application heavy with Ajax requests is put under load, the server response times increase, this may result in all Ajax-requests becoming slower, therefore making the website extremely unfriendly to the end-user. When you just run a JMeter script, this will hardly be noticed, and if you do notice it, you cannot express the impact it will have on the user. You can merely speculate about it.
  • Impact of server-side load on functional behavior;
    Given that the server is experiencing increased load and therefore the business-logic of the application is working hard to handle all requests effectively, it can be safe to say the underlying database may also be stressed and therefore responding slower than expected. Slower response times of both application-logic and database requests can result in buggy behaviour of the application. For example incomplete data returned, or worse, a time-out on data or application-logic. How does the application deal with that? How are these errors reported to the end-user? Will the application still function normally within the browser when certain aspects of the application platform are malfunctioning? The best answer to this is by testing the functionality thoroughly while the application is under load. An easy way to test this repeatedly and consistently is by automating these functional tests, for example using (part of) the automated regression test while the servers are under increased load.
  • Advantage of screenshots of fully rendered pages and possible errors with the application under load;
    As a result of the two points mentioned above, it may be extremely useful for both developers/system engineers and your customer to see errors on the pages affected by the increased server load, such as stylesheets not loading or not loading properly, JavaScript not loading, images missing etc. Screenshots (or screencaptures in movie format) will help make clear to the customer what the problem is and more importantly how big the impact on the end-user will be.

I have listed 3 reasons why combining JMeter and WebDriver can be a good idea. I’d love to hear your suggestions of more reasons to want to combine the two.

In a follow-up post I will go into more detail on ways to achieve an effective combination of JMeter and WebDriver running along side each other, well timed and generating consistent logging and results.

Technical skills are a must for modern testers

In a previous post I already asked “where are the technically strong testers these days?“, in this post I would like to revisit that subject.

When looking at the current developments within IT the focus of testing is getting to be more and more on the business-logic and functionality rather than user-flows. More and more systems are delivered as API-based systems or service oriented architectures where the UI, or front-end, is only to a lesser extend important. UI’s can be put live into the world a lot easier, with béta testing, testing in production etc. Whereas the business-logic of an application, as well as the API’s are increasingly becoming more and more important, since that is where the actual value of most applications is.

traditional testingWhat does this mean for traditional testers? There already is a lot of talk about the changing role of testmanagers and coordinators in the future, what with Agile and things. But I believe the role of the ‘traditional functional tester’ will also lessen.

More and more testing will be pulled into the “technical” side of software, so into the services and API testing spheres, running functional tests through the API rather than through the user interface. This will result in testers, rather than being able to click their way through a user interface, will need to get used to working their ways through systems architectures and learn how to quickly and easily manipulate XML and other formats over a wide variety of protocols from one system to another. They will need to learn how to set up a full end to end chain and troubleshoot within this chain whenever an issue seems to appear.

Out of all the traditional functional testers I personally know, I am not sure what percentage will be able to make this transition, either due to lack of technical insight and understanding or to a simple lack of willingness to invest the time to learn some extra technical skills.

There are of course courses and trainings for testers to gain knowledge in the technical side of software testing, but whether that is enough.. I personally doubt it.
To be or become a good tester with technical knowledge and skills you will need to not only have an understanding of how systems are built up, but also a base understanding of programming. In order to gain these skills I believe more than formal training is required, you need to be willing to invest some extra time in it by home studying these concepts, practicing with them and keeping up with new technologies which might come your way sooner than you’d think.

What’s in a name? The job title paradox

Test consultant

As a test consultant I get hired in lots of different environments and companies, which results in all kinds of interesting job titles I receive. This led me to start thinking about what the title of a job actually means and what the difference is between them. Why should I be a QA Engineer, Test engineer, technical tester, test automation developer, load tester, developing qa engineer or whatever else people can come up with. And why should I care what they call me, as long as I get to do my job and I do it well?

What is the difference between them, why would you call yourself a technical tester or a test automation developer or a test automation engineer for that matter?

Test professional

Within our company we are currently expanding the group of “professionals who are very capable in test automation or load and performance testing”, this raises the question so now and again of what to put on a business card, or an email signature.

Then today I read this (long) article on Harvard Business Review which sounds completely logical, I too in the past have often looked at what the title was on a resume before even reading what the job itself entailed.  At one of my past customers a group of people had the title “test managers” and instead of doing the traditional job of a test manager they were fully dedicated to executing product risk analysis and based on that give advice to other groups within the organisation. At some point during my assignment there, they started a search for a new “test manager” and found they had incredible issues finding someone who matched the job, since all applicants applied to the title rather than the job description. Those who did apply to the job description seemed to lack the “test manager” type role in their resume and thus were not interviewed. It took the company a while to realize this gap and change the name of the role to something more clearly descriptive of what the people in this group did “test risk managers”.

What the above story illustrates to me is that a job title should be descriptive of what you do, not what you want people to think you do.


A former colleague of mine recently received a promotion to something like Senior Vice President of something or other. I am very happy for him that he has a new title, however the job he does is still exactly the same as the one he did 2 years ago: he manages a group of developers and testers. So why not just call yourself that?

Performance tester

In my current assignment I have had several different roles, one being Test Coordinator, which effectively meant I was indeed coordinating the different parties involved with the tests to be executed. However when I switched to a different project within this organisation I started working fully on load and performance testing. The test manager raised the question on what my title should be and I proposed to adjust it to exactly state what I am doing for them at the moment: performance tester.

Getting back to the opening of this post, it is never really clear to me what the difference effectively is between a tester, a test engineer or a QA engineer. In the end, in my experience at least, they all do the same thing: they test software. So why not just call yourself a Software Tester? This makes it abundantly clear to potential colleagues and employers what you do for a living, it is a clear description for recruiters to find a person on and not send some highly inappropriate job offer (which they no doubt will still be doing cause they generally don’t read resume’s of people they contact). And most importantly, it clearly states what you are doing and thus helps you with a direct reality check whether the job you should be doing according to your title is indeed the job you are doing.

My point

So my view on job titles seems clear: just make it state what you do, what your charter within the organisation is and try to keep it clear, short and simple. This makes your life easier as well as that of your colleagues!

Requirements for Load & Performance testing

In my current assignment I am now tasked with writing a performance test plan for testing the performance of an ERP system. In this case, with performance, I mean the actual user experience which needs to be measured. So how long does it take for a screen to be fully rendered within a desktop application for a certain user-type with specific authentication and authorization.

Test data

Considering the organization I am still working in at the moment, the test data will also be an interesting challenge  the requirements we have for data are fairly simple, at first glance at least:

  • 80.000 active users within the ERP
  • 150.000 inactive users within the ERP

However when we start looking at the specifics of how this data is to be built, it becomes a bit more complicated. These users are divided over two organisations, where one organisation is a relatively simple pyramid structure, however the other organisation consists of a huge set (300+) of separate, smaller organisations, which have very flat organizational structures.
Generating this data is going to be fun! Especially since these users will need to be actual active users within the system, with an employment history, since the execution of the performance tests needs historical data to be available, not to forget that the users need to login during the performance tests and actively generate some load on the ERP system.

Performance requirements

Next up is the actual requirements we will be testing for. Some of the questions which popped into my head were:

  • How do you come up with proper L&P requirements?
  • What are bad requirements?
  • How do you get your requirements SMART?
  • How do you then measure these requirements?

So, we had several sessions with the end-users of the system to get to a basic understanding of what they required the application to follow and some really nice requirements came up, for example:

The application should finish a batch-job for at least 500 analyses within a time frame of 8 hours

Considering what SMART stands for this requirement leaves some gaping holes. Sure, it is Timely. The others however are not quite met yet.

This is merely one of many requirements we had to go through to make them SMART. The challenge in the requirements was mainly getting clear to the requirements owners what the difference is, from a testing and specifically performance testing point of view between the original requirements and the actual SMART version of that same requirement.

The example as stated above ended up as the following requirement:

The batch-job for executing predefined data analyses has to finish processing 500 separate analyses within a nightly run of 8 hours, after which the analyses  results are successfully uploaded to the end-user dashboards.

Getting requirements clear for batch-jobs however, is not the most difficult part. The main issue was getting the requirements clear for the user interactions and separating the desktop client interactions from the web-interface.

How do you explain in layman’s terms why a desktop application will by design respond faster than an average web-application and thus that you need different specifications for the two? How do you make clear, again in layman’s terms, that setting up the performance tests for a web-application will not make the scripts reusable for the desktop application, despite them having identical functionality, or even look and feel?

Those are some of the questions I have been struggling with the last few days while writing the performance testplan and with that defining and refining the requirements. (Why is it by the way that I, as the performance tester and defining the requirements I need to test for??)