JMeter Tips & Tricks – Tip 10

Tip 10 – Installing Jmeter-Plugins from commandline

So, you’ve built a beautiful script, now you want to run it from one or more remote servers. These servers are setup, you downloaded the JMeter zit, unzipped it and all. Since you will be running off a server there is no big fat GUI to install the oh so needed JMeter Plugins though.

What do you do to make those work?

Thankfully since version 3.* this has become fairly easy. The JMeter Plugin Manager was introduced and with that comes the PluginsManagerCMD (be it the .bat or the .sh)

When running the PluginsManagerCMD however it is not very friendly in its messaging:

./PluginsManagerCMD.sh 
ERROR StatusLogger No log4j2 configuration file found. Using default configuration: logging only errors to the console. Set system property 'org.apache.logging.log4j.simplelog.StatusLogger.level' to TRACE to show Log4j2 internal initialization logging.
Options for tool 'PluginManagerCMD': <command> <paramstr> where <command> is one of: help, status, available, upgrades, install, install-all-except, uninstall.
ERROR: java.lang.IllegalArgumentException: Command parameter is missing
*** Problem's technical details go below ***
Home directory was detected as: /run/media/martijndevrieze/Data/Downloads/builds/jmeter-nightly/apache-jmeter-r1802079/lib
Exception in thread "main" java.lang.IllegalArgumentException: Command parameter is missing
   at org.jmeterplugins.repository.PluginManagerCMD.processParams(PluginManagerCMD.java:20)
   at kg.apc.cmdtools.PluginsCMD.processParams(PluginsCMD.java:66)
   at kg.apc.cmdtools.PluginsCMD.processParams(PluginsCMD.java:22)
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   at java.lang.reflect.Method.invoke(Method.java:498)
   at kg.apc.cmd.UniversalRunner.main(UniversalRunner.java:105)

So, what exactly are you supposed to do with this script?

Somewhere in between the errors is a useage message:

Options for tool 'PluginManagerCMD': <command> <paramstr> where <command> is one of: help, status, available, upgrades, install, install-all-except, uninstall.

As it turns out, you can get this info with a bit less error information when running the simple command:

./PluginsManagerCMD.sh help

I started out with checking the status of the PluginsManager:

./PluginsManagerCMD.sh status
ERROR StatusLogger No log4j2 configuration file found. Using default configuration: logging only errors to the console. Set system property 'org.apache.logging.log4j.simplelog.StatusLogger.level' to TRACE to show Log4j2 internal initialization logging.
[jpgc-plugins-manager=0.12, jmeter-core=r1802079, jmeter-ftp=r1802079, jmeter-http=r1802079, jmeter-jdbc=r1802079, jmeter-jms=r1802079, jmeter-junit=r1802079, jmeter-java=r1802079, jmeter-ldap=r1802079, jmeter-mail=r1802079, jmeter-mongodb=r1802079, jmeter-native=r1802079, jmeter-tcp=r1802079, jmeter-components=r1802079]

We now have an overview of the plugins installed in JMeter, nothing out of the standard packaged set.

Now let’s install some plugins, for that the following command is needed:

./PluginsManagerCMD.sh install  <paramstr>

Where the <paramstr> needs to be filled with something sensible. This something sensible is the ID of the plugin you want to install.

You can find the ID’s on the Jmeter-Plugins site in the menu item “Browse plugins”. Look at the plugin(s) you need and grab the accompanying ID:

jmeterpluginID

For the sake of this example the command will be:

./PluginsManagerCMD.sh install websocket-samplers

Unfortunately JMeter will not provide you any feedback while running this. The only easy way to verify that the plugin is there is by either running the script that needs this plugin, or checking your jmeter lib/ext directory:

ls ../lib/ext/
ApacheJMeter_components.jar ApacheJMeter_http.jar ApacheJMeter_junit.jar ApacheJMeter_native.jar readme.txt
ApacheJMeter_core.jar ApacheJMeter_java.jar ApacheJMeter_ldap.jar ApacheJMeter_tcp.jar
ApacheJMeter_ftp.jar ApacheJMeter_jdbc.jar ApacheJMeter_mail.jar jmeter-plugins-manager-0.12.jar
ApacheJMeter_functions.jar ApacheJMeter_jms.jar ApacheJMeter_mongodb.jar JMeterWebSocketSamplers-0.10.jar

 

 

 

 

 

 

 

Jmeter Tips & Tricks – Tip 9

Tip 9 – Generating a report from your log file

When running a performance test with Jmeter it is generally adviced to run the test  in non-gui mode and to log your responses to a file. My typical command for a performance test looks something like this:

Jmeter –n –t TestScenario.jmx –j jmeter-TestRun01.log –l yyyyMMdd-TestRun-10000Threads-300TPS.jtl

Where the commandline flags have the following meaning:

-n => Non-GUI

-t => the testscenario JMX file to run as  a test

-j => where to write the Jmeter logfile

-l => where to write the sample results to. This typically gets a JTL-extension

Once you have run your test successfully the real work of a performance tester starts, analyzing the outcomes and communicating the results and of course providing advice what to do with these results.

For the result-graphs you can of course use the Jmeter Listeners. You can also use Excel. But neither are a very easy nor a friendly way to do it.

Generally speaking a bit of JTL log is not very friendly to read:

timeStamp,elapsed,label,responseCode,responseMessage,threadName,dataType,success,failureMessage,bytes,sentBytes,grpThreads,allThreads,URL,Latency,Encodi
ng,SampleCount,ErrorCount,IdleTime,Connect
1504276591952,296,GET - Login screen,200,OK,jp@gc - Ultimate Thread Group - EA-OP 1-100,text,true,,523937,1116,1,1,https://poc-15.educus.nl/app/login,10
5,UTF-8,1,0,0,86
1504276592251,259,GET - Login screen,200,OK,jp@gc - Ultimate Thread Group - EA-OP 1-99,text,true,,523937,1116,2,2,https://poc-15.educus.nl/app/login,93,
UTF-8,1,0,0,77
1504276592552,282,GET - Login screen,200,OK,jp@gc - Ultimate Thread Group - EA-OP 1-98,text,true,,523937,1116,3,3,https://poc-15.educus.nl/app/login,98,
UTF-8,1,0,0,80
1504276592851,292,GET - Login screen,200,OK,jp@gc - Ultimate Thread Group - EA-OP 1-97,text,true,,523937,1116,4,4,https://poc-15.educus.nl/app/login,105
,UTF-8,1,0,0,88
1504276593151,254,GET - Login screen,200,OK,jp@gc - Ultimate Thread Group - EA-OP 1-96,text,true,,523937,1116,5,5,https://poc-15.educus.nl/app/login,97,
UTF-8,1,0,0,80
1504276593451,212,GET - Login screen,200,OK,jp@gc - Ultimate Thread Group - EA-OP 1-95,text,true,,523937,1116,6,6,https://poc-15.educus.nl/app/login,87,
UTF-8,1,0,0,72
1504276593751,225,GET - Login screen,200,OK,jp@gc - Ultimate Thread Group - EA-OP 1-94,text,true,,523937,1116,7,7,https://poc-15.educus.nl/app/login,89,
UTF-8,1,0,0,73
1504276594051,210,GET - Login screen,200,OK,jp@gc - Ultimate Thread Group - EA-OP 1-93,text,true,,523937,1116,8,8,https://poc-15.educus.nl/app/login,90,
UTF-8,1,0,0,74
1504276594351,214,GET - Login screen,200,OK,jp@gc - Ultimate Thread Group - EA-OP 1-92,text,true,,523937,1116,9,9,https://poc-15.educus.nl/app/login,88,
UTF-8,1,0,0,73
1504276594651,228,GET - Login screen,200,OK,jp@gc - Ultimate Thread Group - EA-OP 1-91,text,true,,523937,1116,10,10,https://poc-15.educus.nl/app/login,9
1,UTF-8,1,0,0,75

Thankfully JMeter has a very nice, although not very elegant solution to this. I consider this not very elegant since you can only trigger it via commandline. The results however are quite elegant and pretty to view.

Run the following command to get your pretty report once your test is finished:

jmeter -g yyyyMMdd-TestRun-10000Threads-300TPS.jtl -o WriteThisToACleanDirectory

This generates a very nice HTML/js based reporting dashboard. I will refrain from going into details about how nice the dashboard has become over the years, you can read all that is in the dashboards on the Apache Jmeter site.

The landing page may look something like this:

The graphs on the dashboard are all ineractive, you can zoom in on specific details, filter out specifi requests etc. I really like what has become of these reports.

The negative side of that way of generating a report is that you still have to do it once you are done running your performance tests.

That too can be solved! When running tests from a commandline I generally use a command close to this:

jmeter -n -t TestScenario.jmx -j jmeter-TestRun01.log -l yyyyMMdd-TestRun-10000Threads-300TPS.jtl -e -o OUTPUTDIRECTORY > /dev/null 2>&1 &

The addition of the dashboard generation is done with the

-e -o OUTPUTDIRECTORY

flags and arguments. The little extra sauce I give is that I generally open a second console where I tail the JMeter log and potentially the JTL log. So in my main window, where I started the testrun, I prefer to have my commandline available to do useful things such as shutdown JMeter if so required. Hence I send the console output to

/dev/null

and send any possible error stream directly to the output stream (which again is sent into the void that is /dev/null

Last but not least I background the process with the & in order for me to have my console back and available.

 

Jmeter Tips & Tricks – Tip 8

Tip 8 – Generating a specific amount of hits per second

JMeter is generally oriented towards a performance test approach where the load is based on a specific set of concurrent users, or threads. When talking metrics with systems engineers however, you will generally hear something more towards hits per second, requests per second or transactions per second. So how do you get JMeter to generate a certain amount of hits per second?

There are of course several ways to go about this, but in this article I will limit myself to a fairly simple method, using a Timer.

Constant Throughput Timer

constant throughput timer
The Constant Throughput Timer can be very useful in generating a (surprise!!) constant throughput.

What this timer does, is make sure that, regardless of the amount of threads you have started, the test will pause whenever needed to throttle the amount of requests per second. It is good to note by the way, that the timer is NOT based on milliseconds or seconds, but instead is counting per minute.

When your requirements state that the application (and server) should manage to survive some 60 hits/second, you will need to calculate your hits per second back to the actual amount of hits per minute (e.g. 10 hits/second * 60 seconds = 600 hits/min).

Keep in mind that there may be a difference in your load requirements, you have to really dig up from your customer/product owner or whoever came up with the performance requirements what exactly they expect. When they define hits per second, what do they mean with that? is that pageviews or is that actual requests (e.g. 1 page can consist of more requests for HTML, CSS, JS, Images etc.). Always verify and double check that what you mean with hits per second or requests per second is indeed what they also mean!

Understanding the “Calculate Throughput based on” variable constant throughput timer

There are several ways the throughput can be calculated and enforced. The default setting is “this thread only“, in my eyes however the most logical setting (based on the above requirements) is the “all active threads” setting.

  • this thread only – each thread, as defined in your Thread Group thread properties, will try to stick to the target throughput. This means that when you have 150 threads, your throughput will be 150 * Target throughput.
  • all active threads in current thread group – the target throughput is divided across the active threads in the thread group. In other words, this will give you the actual target throughput as you have configured. This throughput is for this specific thread group only! Threads themselves are delayed and started based on when this particular thread last ran. e.g.
  • all active threads – When you have more than one thread group, this setting becomes interesting. This will divide the target throughput across all active threads in all Thread Groups. Be aware, each Thread Group requires a Constant Throughput timer with the same settings for this to work.
  • all active threads in current thread group (shared) – Each thread is delayed based on when any thread in the group last ran, meaning the threads run consecutively rather than concurrently. For the rest this setting does exactly the same as the “all active threads in current threadgroup”, e.g. this will give you the actual target throughput as you have configured.
  • all active threads (shared) – Each thread is delayed based on when any thread in the group last ran, meaning the threads run consecutively rather than concurrently. Any thread here has again a wider meaning than in the previous setting, this setting runs across all threads and thread groups you have configured.

How do you know which setting you need?

These different settings can be quite confusing to any Jmeter user, even to experienced users. I would therefore recommend the following:

Make sure you put the constant throughput timer in the root of your testplan (e.g. at the highest level) and let it dictate the throughput of all of your threads and thread groups, e.g. “all active threads“. That way you know for sure what the actual throughput if your test is.

In the case of a somewhat complex environment, where you have several thread groups with each different amounts of requests per second, make sure you set the timer within the root of that particular thread group and stick to the “all threads in current thread group“.

Jmeter Tips & Tricks – Tip 6

Tip 6 – Effective use of Transaction Controllers

Whether you are trying to script a website, a webservice or something else, quite often you find that one functional action results in several requests to a server.

Take for example an average login-request seen from a functional point of view:

LoginScreenA user types in username & password and hits the Login button.
If the username/password combination was valid the user is than shown a dashboard.

This seems straight forward enough from a functional point of view. What actually happens however is a bit more than just this one request:

  • Username/password are sent to an authentication service
  • Authentication service sends an authenticated message back (assuming you are indeed authenticated by the application).
  • The authentication message contains a redirect away from the authentication service to the dashboard a user expects. This redirect contains information such as the AuthToken that was just recieved from the service.
  • Subsequent requests for the Dashboard-contents are sent.

In a simple setup, this may require only 1 request in Jmeter, just the login. Since Jmeter can take care of any redirecting itself.

Making Jmeter responsible for blindly following all redirects however, does not always give you the performance metrics you may want or need. When you need to know exactly which part of the login sequence is giving your users some form of grief, you may want to build these requests completely yourself in Jmeter. That will give you the possibility to see if there is high latency, big loads etc. on a per-request basis.

You may end up with something like this: jmeter-4requests

The 4 requests you see in this example are all separate requests used to log a user into the system and get the base information for the first screen of the application.

This sequence could have been captured in 1 request, Login, since the other requests are handled by HTTP/302 responses.

By building the requests yourself and clustering them together in a Logic Controller you now can see and measure how long the entire login request, step by step, takes. The Logic Controller I have used here is a Transaction Controller, this controller can add all child-transactions into one single response time for you by setting the appropriate flag. TransactionController
When set, “Generate parent sample” will ensure that your statistics have a single statistic for the complete set of requests, yet you can easily gain information on the separate requests while running a loadtest by unchecking this flag and looking into each step by itself as well.

This gives you the possiblity to find out which of the steps slows the entire transaction of logging in to your application (or whichever functional step you define in a Transaction Controller of course) .

Jmeter Tips & Tricks – Tip 5

Tip 5 – Logging made easy

Generally speaking it is useful to write your logs to file when running tests. This will help reproduce your results later on, make life easy with comparing results of different tests and will serve as a useful audit log for your load report later on.
The easy way to save the log is to add a filename string in the “Write results to file” box of your listener like this:

ViewResultsTreeWriteToFile

As you can see in this example I generally specify the logname to be at least a bit meaningful, e.g. which Listener did I use and how many threads did I start (the latter is not visible in this sample).

You do however quite regularly run the same amount of threads more than once, so how do you make clear which run this was? Updating the filename very quickly becomes an nuisance. So why not automate that?

I generally use the ${__time()} function. Making the filename something like this:
${__time(yyyyMMdd-HHmmss)}-LISTENERNAME-NUMBEROFTHREADS.jtl

ViewResultsInTableWriteToFile

This kind of logging results in a set of files which is easily sortable on filename and easily readable based on the amount of threads started for this particular test:

20160404-103706-resultsInTable-10threads.jtl

Jmeter Tips & Tricks: Tip 3

Tip 3: URL Encoding

This tip is a fairly simple one, but I often use it when building scripts.

URL encoding is quite often needed in heavier webapplications on the variables you extract with the Regular Expression extractor or other extractors.

How do you make this variable URL Encoded when not using the standard “Parameters” but you need to use the Body Data instead?

The given variable we are going to encode is something I see rather too often, a ViewState. In order to grab this thing from the page I have the following regular expression:

Reference Name: VIEWSTATE
Regular Expression: id="__VIEWSTATE" value=(.+?)"

So, now we have a variable ${VIEWSTATE} which needs URL Encoding. The way to do this, within the Body Data is as follows:

${__urlencode(${VIEWSTATE})}

So whatever your variable is, just put it in between the brackets in the ${__urlencode()} and you have your encoded variable.

 

Jmeter tips & tricks – Tip 1

Over the years I have taught myself a few simple tips & tricks in Jmeter and I thought it might be nice to share them here. Some are simple, some are a bit more complex but I hope all of them will be useful to reuse for anyone. Where possible I will see if I can add an actual JMX file to download for your ease of use.

Tip 1: Crawling URLs

A URL crawler can be used for several things such as:

  1. verifying all URL’s on the site
  2. mimicking random browse behaviour and thus generating random load on the server

Setting up a crawler involves a few steps. First off you will need a thread group, as you need it with everything. Within that thread group you place an HTTP Request going to your base-page, usually the homepage of the web application.

Then the fun part starts, now we add a While Controller with a simple condition added to it:

LAST - exit loop when last sample in loop fails. If the last sample just before the loop failed, don't enter loop.

You can also make it with an empty Condition, however I prefer using this one so that if your script didn’t already break on an error, it will absolutely break on an error before hitting the loop.

Within the While Controller we now place another HTTP Request, where the path is set to

.*

Underneath the HTTP Request now add an HTML Link Parser.

You now should have a structure similar to this:

TestPlan
  |__ Thread Group
             |__ HTTP Request
             |__ While Controller
                       |__ HTTP Request
                                 |__ HTML Link Parser

The Link Parser does nothing other than extract links from the received HTML.

Next up is adding an If Controller to ensure the link we just grabbed indeed is working and if not, send the crawler back to the base page (e.g. the home page you defined in the first HTTP Request).
The If Controller needs some Conditions in order to do this. My preferred option is to add the following bit of JavaScript:

${__javaScript(!${JMeterThread.last_sample_ok})}

As a child node underneath this If Controller you now need to hang yet another HTTP Request, which sends the user back to your initial homepage. When filling in URL’s etc. please keep in mind that you should just keep life simple and make URL’s either read from a CSV file or simply add them as a global variable in the Test Plan.

The entire plan now should look like this:

TestPlan
  |__ Thread Group
       |__ HTTP Request > Home page
       |__ While Controller > Condition "LAST"
                 |__ HTTP Request > Path ".*"
                           |__ HTML Link Parser
                 |__ If Controller > Condition "JavaScript"
                           |__ HTTP Request > Back to Home page

Please be careful in using this script, this might lead to an unintended Denial Of Service attack if ran uncontrolled and with high load!

A sample JMX of this simple version of a web crawler can be found here.