Jmeter Tips & Tricks – Tip 7

Tip 7 – adjusting the JVM HEAP size, why and how to

When running JMeter tests, it may happen that the test is not successful and JMeter crashes with an OutOfMemoryError. This can happen due to all kinds of things, most of the time however you are trying to generate too many threads with an insufficient amount of memory allocated to your Java Virtual Machine. More information on what the HEAP is, how the JVM deals with the HEAP can be found here.

The HEAP size is defined on startup of your java application, in our case when we start JMeter. In other words, adjusting the HEAP means we will need to adjust the startup script. The startup script can be found in your JMeter “bin” directory. In my case, on a linux environment, it can be found here:

/opt/apache-jmeter-3.1/bin

When opening the “jmeter” file in your favorite editor, scroll down until you hit a line stating the following (on Windows based systems open “jmeter.bat”):

# This is the base heap size -- you may increase or decrease it to fit your
# system's memory availability:
HEAP="-Xms512m -Xmx512m"

On Windows systems the line looks slightly different, in the jmeter.bat, due to the nature of BATCH files and DOS working differently from Linux systems

set HEAP=-Xms512m -Xmx512m

When you adjust this line to read something like this:

# This is the base heap size -- you may increase or decrease it to fit your
# system's memory availability:
HEAP="-Xms1024m -Xmx4096m"

the HEAP size will by default be larger, meaning you do not have to think about it anymore when starting JMeter. This does pose one possible challenge, it slows down the startup of JMeter somewhat (at least on most machines I have done it). This seems to be because the JVM will first allocate all the memory you defined in the Xms param, ensuring the JVM is at the appropriate size instantly.

So, we changed the values of Xms and Xmx, what do these two mean?

-Xms ==> minimum heap size. This is the value JAVA will attempt to claim in order to run JMeter. This does NOT imply that it immediately gets this fully allocated by the OS. The OS typically doesn't give an application all blocks it asks for until really needed.
-Xmx ==> maximum heap size. This is the value JAVA will use as a maximum size of the JVM. When JMeter passes this level, an OutOfMemoryError will occur once more.

An alternative, more flexible setup to change the heapsize is to pass the JVM arguments on the commandline when starting JMeter. This of course assumes you are comfortable in commandline and start JMeter from commandline.

[user@machine ]$ JVM_ARGS="-Xms1024m -Xmx2048m" jmeter

Be Aware!

Please always make sure you leave enough memory for your OS and other applications to run on your machine when allocating more HEAP space to a JVM. Not leaving enough memory for your OS to keep running will result in your machine simply being non-responsive and thus your test and testresults being rendered useless.

 

5 common reasons for a penetration test

windowshackingI am slowly but steadily moving more and more into software security. This has led me to have regular discussions on the reasoning behind doing a penetration test on an application. “Why do did  you decide to have your application subjected to a penetration test?” Is a question I generally ask my (prospective) customers. Most of the times I cannot get a clear answer on this question. Answers vary between “I want to ensure my application is safe” to “We are required to do it for ISO certification”.

The most common reasons for requesting a penetration test can be summarized in the following categories:

  • Peace of mind – we have done what we can to make sure the data is safe from our side
  • Image – our company image will be severely damaged if we are hacked or have security issues on the application
  • Financial risks – we store sensitive data, it will cost of dearly if that data is compromised in some way
  • Certification – we want to attain ISO27001 certification and need regular penetration tests for that
  • Trust – we have bought a software solution and want to make sure it is not open to abuse. But also on the vendor side: we want to be able to tell (and show) our customers that our software is safe and not open to abuse or data leaking

Before I agree to start working on a penetration test on your application or environment I want to know what the real reasons for you asking for a penetration test are.

Why do I want to know the reasoning behind a request for a penetration test?

The answer is quite simple, I want to understand which risk (if any) you are trying to cover when requesting a penetration test. In order to make sure the report I write after a test actually is valuable for you, my client, I want to grasp what you are afraid off.
I already mentioned the common reasons, however these reasons all have at least one type of risk behind them you, my client, are worried about.
If for example you fear your end-users might want to game your software in some way to their benefit (think of a student tracking system, which also includes their grades) we can focus the penetration test on inventorising and possibly mitigating that particular risk.

In several follow up posts I will go into some detail how a penetration test can actually help.

Jmeter Tips & Tricks – Tip 6

Tip 6 – Effective use of Transaction Controllers

Whether you are trying to script a website, a webservice or something else, quite often you find that one functional action results in several requests to a server.

Take for example an average login-request seen from a functional point of view:

LoginScreenA user types in username & password and hits the Login button.
If the username/password combination was valid the user is than shown a dashboard.

This seems straight forward enough from a functional point of view. What actually happens however is a bit more than just this one request:

  • Username/password are sent to an authentication service
  • Authentication service sends an authenticated message back (assuming you are indeed authenticated by the application).
  • The authentication message contains a redirect away from the authentication service to the dashboard a user expects. This redirect contains information such as the AuthToken that was just recieved from the service.
  • Subsequent requests for the Dashboard-contents are sent.

In a simple setup, this may require only 1 request in Jmeter, just the login. Since Jmeter can take care of any redirecting itself.

Making Jmeter responsible for blindly following all redirects however, does not always give you the performance metrics you may want or need. When you need to know exactly which part of the login sequence is giving your users some form of grief, you may want to build these requests completely yourself in Jmeter. That will give you the possibility to see if there is high latency, big loads etc. on a per-request basis.

You may end up with something like this: jmeter-4requests

The 4 requests you see in this example are all separate requests used to log a user into the system and get the base information for the first screen of the application.

This sequence could have been captured in 1 request, Login, since the other requests are handled by HTTP/302 responses.

By building the requests yourself and clustering them together in a Logic Controller you now can see and measure how long the entire login request, step by step, takes. The Logic Controller I have used here is a Transaction Controller, this controller can add all child-transactions into one single response time for you by setting the appropriate flag. TransactionController
When set, “Generate parent sample” will ensure that your statistics have a single statistic for the complete set of requests, yet you can easily gain information on the separate requests while running a loadtest by unchecking this flag and looking into each step by itself as well.

This gives you the possiblity to find out which of the steps slows the entire transaction of logging in to your application (or whichever functional step you define in a Transaction Controller of course) .

Jmeter Tips & Tricks – Tip 5

Tip 5 – Logging made easy

Generally speaking it is useful to write your logs to file when running tests. This will help reproduce your results later on, make life easy with comparing results of different tests and will serve as a useful audit log for your load report later on.
The easy way to save the log is to add a filename string in the “Write results to file” box of your listener like this:

ViewResultsTreeWriteToFile

As you can see in this example I generally specify the logname to be at least a bit meaningful, e.g. which Listener did I use and how many threads did I start (the latter is not visible in this sample).

You do however quite regularly run the same amount of threads more than once, so how do you make clear which run this was? Updating the filename very quickly becomes an nuisance. So why not automate that?

I generally use the ${__time()} function. Making the filename something like this:
${__time(yyyyMMdd-HHmmss)}-LISTENERNAME-NUMBEROFTHREADS.jtl

ViewResultsInTableWriteToFile

This kind of logging results in a set of files which is easily sortable on filename and easily readable based on the amount of threads started for this particular test:

20160404-103706-resultsInTable-10threads.jtl

Jmeter Tips & Tricks – tip 4

Tip 4: grabbing a new value for one variable across requests

Imagine the situation, you have this annoying value popping up on every response, which you need in every subsequent request. For example something like a VIEWSTATE, which is quite common and really annoying. If you lose it or pick up the wrong (one request older) version, your request is invalid and your session broken.

So, the simplistic way to grab that thing is on every request add something like the Regular Expression Extractor with a regExp in it grabbing the correct VIEWSTATE. This however ends up being a bit of a mucky business, because you end up with a lot of RegExp Extractors all over the place, resulting in a very complex JMX file.

The tip

regularexpressionextractorPut the Regular Expression Extractor with all stuff in the right places right at the top of your Thread Group. This will ensure that the Extractor is triggered on each and every request sampler, and thus you always have the most up to date (and correct) version of your variable.

 

Jmeter Tips & Tricks: Tip 3

Tip 3: URL Encoding

This tip is a fairly simple one, but I often use it when building scripts.

URL encoding is quite often needed in heavier webapplications on the variables you extract with the Regular Expression extractor or other extractors.

How do you make this variable URL Encoded when not using the standard “Parameters” but you need to use the Body Data instead?

The given variable we are going to encode is something I see rather too often, a ViewState. In order to grab this thing from the page I have the following regular expression:

Reference Name: VIEWSTATE
Regular Expression: id="__VIEWSTATE" value=(.+?)"

So, now we have a variable ${VIEWSTATE} which needs URL Encoding. The way to do this, within the Body Data is as follows:

${__urlencode(${VIEWSTATE})}

So whatever your variable is, just put it in between the brackets in the ${__urlencode()} and you have your encoded variable.

 

Jmeter Tips & Tricks – Tip 2

Tip 2: Semi-random think time from a user

First of all, adding think-time to a Jmeter script can be done with a host different types of timers. The version I am going to propagate here works well for me, I have the feeling I have proper control over the timer this way and I can predict what it does. This does not imply that using a different way of adding think time is wrong in my view. I just happen to like this version.

My think time consists of two pieces: a Test Action with a Uniform Random Timer attached to it:

|__ Test Action > Action: Pause, Current Thread
   |__ Uniform Random Timer > Random Delay Max & Constant Delay Offset set

So, what do the Test Action and the Uniform Random Timer do?

The Test Action sampler is simply a sampler you can use to make a script wait for a specified amount of time.

The Uniform Random Timer however is a bit less simple to explain and understand:

The apache site states the following:

This timer pauses each thread request for a random amount of time, with each time interval having the same probability of occurring. The total delay is the sum of the random value and the offset value.

So what that means is the following.

uniform_random_timer

When setting the Random Delay Maximum, the timer will pick any random value between 0 and 2000. Now add to that the Constant Delay Offset and you have the actual wait (or pause) time JMeter will use in the script in between the requests.

 

So in this example the value will be somewhere between 1 and 3 seconds.

Disadvantage of this method: You will need to set this think time around every HTTP Request you want to have think time.

The functions as described in this post as well as in the previous Jmeter Tip can be found in this JMX file.