JMeter and remote servers – a tutorial

In my previous post I discussed why you might want to run your own servers for load & performance testing. In this article I will elaborate a bit on how to setup your machines. The Apache Jmeter pages of course have an explanation on how to setup remote tests. What I have heard from colleagues and what I experienced myself, is that that explanation is not always as clear and concise as may be preferred. This is my attempt at giving a readable explanation on how to setup JMeter with remote instances.

Some basic terminology

I will use the terms Local and Remote to identify the different sides of the configuration needed to get things working.

  • Local is to be read as your workstation, e.g. the machine you use to build your JMeter scripts.
  • Remote should be considered any machine that will be running a (headless) JMeter instance and will help generate load on an object.

jmeter Client - Service image

How to setup JMeter locally to work with remote machines

Configuring JMeter to work with your own (or your customers) load generating servers is relatively easy, however it is not as straightforward as one might hope and expect.

Client Configuration

Firstly we need to add the fully qualified domainnames or IP addresses of the servers to JMeter. These names, or addresses will, after starting JMeter show up in the “Run” menu of Jmeter under the “Remote start” item.

In order to do so take your favorite file-editor and open up “jmeter.properties” on your own, local machine and edit the following section:

#---------------------------------------------------------------------------
# Remote hosts and RMI configuration
#---------------------------------------------------------------------------
# Remote Hosts - comma delimited
remote_hosts=<INSERT YOUR OWN IP ADDRESSES OR NAMES, AS A COMMA SEPARATED LIST>
client.rmi.localport=7000

The two items just edited are the settings for the remote hosts, so the IP addresses of the servers you want to use within your grid. This can become a relatively long list of addresses, since you can technically add as many as you desire. When looking at my personal setup I have several instances of the remote_hosts line, where only one is active at any given point in time. Each customer that has it’s own remote machines, will receive a separate line, which is used when working for that particular customer. This is in order for me to make sure I do not accidentally start a test from servers that I should not be using or which are simply not available at the time. The latter mistake is not as damaging as the first, however I consider it annoying because JMeter will hang for a fairly long time attempting to connect to the remote machines.

The second line defines the port on which the local JMeter instance will listen for the Remote Method Invocation or RMI connections from the servers.

Server configuration

RMI ports

In the JMeter properties file (jmeter.properties) you will have to find the following settings and adjust them to your needs. I have chosen port 7000 for the client RMI port and  60000 for the server RMI port since that was a port that was open within most of my customer firewalls and not linked to anything else. Technically these ports can be left to the defaults filled in in the standard jmeter.properties. Keeping the default value for the client RMI port however will often lead to unpredictability due to firewalls being locked on loads of ports.

# Parameter that controls the RMI port used by the RemoteSampleListenerImpl (The Controler)
# Default value is 0 which means port is randomly assigned
# You may need to open Firewall port on the Controller machine
client.rmi.localport=7000

# To use a specific port for the JMeter server engine, define
# the following property before starting the server:
server.rmi.localport=60000

Sample senders

Having set the correct ports to ensure all data gets sent back to the local instance for reporting in your Listeners is step one. The next step is ensuring the data is sent back to your local instance in a way that works best for you. The settings for this can be found in jmeter.properties as well, the part you are looking for is the “Remote batching configuration”

#---------------------------------------------------------------------------
# Remote batching configuration
#---------------------------------------------------------------------------

This part of the configuration may be a bit more sensitive and difficult to get right. There are a lot of different settings to play with depending on the mode you choose. My preferred setting is this:

mode=Standard

This mode sends samples synchronously as soon as they are generated, thus giving you a constant update on the progress of the test.
In order to save bandwidth though, it may also be preferred to store all samples in memory until the end of the run. I prefer to not do this, since this can lead to loss of data is the JMeter instance crashes due to memory issues, but if you want to preserve bandwidth it is a good option as well.

mode=DiskStore

Connect the dots: Portforwarding

In order to easily connect with my remote servers and have them ready for testing I have created a simple set of shell scripts. They look something like this:

ssh -R 7000:127.0.0.1:7000 USER@SERVER.NAME.COM <<EOF 
cd apache-jmeter-2.11/bin/ 
./jmeter-server 
EOF

So, what is hapening here?

ssh -R 7000:127.0.0.1:7000 USER@SERVER.NAME.COM <<EOF

This command will start an SSH reverse tunnel, which connects the remote port 7000 to the local port 7000. This effectively makes it possible for the server to send data back directly to your local JMeter instance, thus ensuring you can see the results of your tests on your own local machine while the test is running.
Adding the <<EOF to ensure the script, once logged in will keep executing commands until it reaches End Of File, where the script exits.

cd apache-jmeter-2.11/bin/

Once logged in to the remote machine, change directory to where the JMeter executable is installed. In this particular example I assume the JMeter instance used is not a default installed binary but a custom version. If you use a standard JMeter install on a default Linux distribution (i.e. Ubuntu) this step can be omitted.

./jmeter-server
EOF

Start the JMeter Server instance and signal to the script that the end of file has been reached and no more commands need to be sent to the remote server. Instead of executing “jmeter-server” it is also possible to start the server with the basic JMeter startup script by adding the -s flag, thus resulting in the following command:

jmeter -s
EOF

You can add these steps for all servers in separate shell scripts or of course make it one big setup script that by default will start all Jmeter-servers within your grid.

Now that the servers have been started it is time to run a test to see if this actually worked.

Starting the test

Starting a test with the remote machines is relatively easy once all of the above has been done. In JMeter hit the Run menu and select either the machine you would like to use or select the Remote Start All to just start all machines.

JMeter Remote Start

A very important thing to keep in mind in running tests on a set of remote machines is the following (as shown on the Jmeter Usermanual page as well):

Note: The same test plan is run by all the servers. JMeter does not distribute the load between servers, each runs the full test plan.

In other words, if you set your testplan to have 500 threads and you run this test on 4 servers you will effectively start not 500 threads spread out over those servers, you will start 500 threads per server thus resulting in 4000 threads.

If you do not want to start the test from the GUI there is also the possibility of starting remote tests from commandline. You can choose to either start on all the servers you have configured:

jmeter -n -t script.jmx -r

Or you can be picky and specify the machines you want to start the tests on:

jmeter -n -t script.jmx -R server1,server2

In my tests I quite often use CSV files to feed data to my tests, for example to feed usersnames and passwords, search terms etc. In order for these CSV files to be picked up on the server side as well, they need to be uploaded to the server. Where JMeter will send the testscript to the server, it does not send any data-files to the servers.

43 thoughts on “JMeter and remote servers – a tutorial

  1. Pingback: Testing Bits – 11/30/14 – 12/06/14 | Testing Curator Blog
  2. Hi,
    mode=Standard send all html content (for every url) from server to client each time a http reponse is receved => good to debug but no heavy load test
    Since JMeter 2.9 the defaut mode is mode=strippedbatch, send non html content but another infos like size, status, elasped time … and wait for n secondes or wait for n requests before send stripped reponse to the client. => good for heavy load test.
    I can’t preconized standard mode.
    Regards.
    Vincent D.

      • Hi Martijn, Thanks for the information. Is it JMeter distribution testing could we able to simulate (with four agents)75,125,150,200 user’s on each machine?

      • the amount of users you give in the JMX is the amount you generate per engine. So that should be no problem I would say.

        If you want to build up the load like that you could use the Jmeter Plugins Ultimage Threatgroup, that gives you a lot more freedome to play with one scenario and running it with different loads without adjusting the JMX every time.

  3. Hi, I set mode to Standard on client and server node, but get nothing back. The Jmeter looks like just send all the requests out and do not wait for response before it returns. I tried to run the same test plan on the server, using local mode. It worked well. So, I have no idea what did I do wrong. Could you plz help me with that?

    • Hey Wesley,
      Do you have the reverse port tunneling started and did you verify that is indeed working as expected?
      If you are not sure your tests are running properly on the server when started remotely you can always SSH into the server and tail your jmeter logs.
      What you describe sounds like the reverse tunnel not sending things back properly.

      • Thanks for reply. I can grep logs from the application server I test after Jmeter -R returned. So I’m sure Jmeter have sent all the request in a very short time. Perhaps I haven’t described my problem accurately. I did get summary back as follow. But what I need is the response from the application server as I get when I run Jmeter in local mode.

        $ ./jmeter -n -t /home/cepm/JMeter/nodes/lt/wesley/apache-jmeter-2.13/test/EFPerformanceTest.jmx -R icam-dev-app3
        Creating summariser
        Created the tree successfully using /home/cepm/JMeter/nodes/lt/wesley/apache-jmeter-2.13/test/EFPerformanceTest.jmx
        Configuring remote engine: icam-dev-app3
        Starting remote engines
        Starting the test @ Tue Sep 22 12:01:47 CST 2015 (1442894507999)
        Remote engines have been started
        Waiting for possible shutdown message on port 4445
        summary = 80 in 7.2s = 11.1/s Avg: 42 Min: 18 Max: 115 Err: 0 (0.00%)
        Tidying up remote @ Tue Sep 22 12:01:56 CST 2015 (1442894516211)
        … end of run

      • Hey Wesley,
        Might be easier to help you through either Skype (martijndevrieze or Google Hangouts (martijndevrieze at qa-rocks.com) in a chat?

        I think I understand what you are looking for but not 100% sure.

      • I appreciate for that. Unfortunately, it is not a convenient time for a chat at the moment. Shall we communicate via email? My email address is yifaxu@cisco.com. I’m looking forward to your help.

  4. This how I configured my environments:

    ON LOCAL in jmeter.properties:

    #—————————————————————————
    # Remote hosts and RMI configuration
    #—————————————————————————

    # Remote Hosts – comma delimited
    remote_hosts=127.0.0.1:55501
    client.rmi.localport=55512
    mode=Batch
    #remote_hosts=localhost:1099,localhost:2010

    ON REMOTE SERVER:
    #—————————————————————————
    # Remote hosts and RMI configuration
    #—————————————————————————

    # Remote Hosts – comma delimited
    server_port=55501
    server.rmi.localhostname=127.0.0.1
    server.rmi.localport=55511

    AND CONNECT FROM LOCAL PC TO REMOTE SERVER:

    ssh -L 55501:127.0.0.1:55501 -L 55511:127.0.0.1:55511 -R 55512:127.0.0.1:55512 username@host “/path/to/yours/jmeter-server -Djava.rmi.server.hostname=127.0.0.1”

    You can do as I have mentioned above if you have done Ssh key auth between your machine and remote server, so remote server won’t ask you for password and will start the Jmeter-server.

  5. Very nice blog, Sir. Thanks for sharing your knowledge on this.

    I have some questions:

    1. You had said ‘In my tests I quite often use CSV files to feed data to my tests, for example to feed usersnames and passwords, search terms etc.’. How do you manage the data used by server1 not used by server2? How do you handle server specific data when the same jmeter test plan is run by all the servers?

    2. jmeter -n -t script.jmx -r -> This is great to start the test on all the remote servers. If i need to stop the test abruptly, If i press the ctrl+c, only the client instance is stopped in non-gui mode. Remote servers are still execute the test. What is the non gui equivalent for ‘Remote Stop All’?

    Thanks Sir.

  6. Hi, I am actually running this, but i get errors saying Waiting for possible shutdown message on port 4445. What can i do for this?

      • Hi Martijn, i am running the Jmeter non-gui call from the master, and waiting for the response from the slave. After the Test has run on the slave, on the master i get a msg stating “Waiting for possible shutdown message on port 4445”. and the execution just stays there for hours.

      • Vishal,
        This week I will be setting up a new server again for JMeter remote testing and I will see if I get what you have. I have not seen this issue before.

        You do get your data back on the client side, right? Seems like the listener is not where you expect it to be.

  7. Hi there Martijn,

    I followed your steps on testing remote server yet I’m getting some error when I tried to do it with ‘Remote Start’. It says ‘Connections refused to host’. I don’t know where I did I missed.
    Btw, FYI, I have these details with my server which I put in jmeter.properties:

    # Parameter that controls the RMI port used by the RemoteSampleListenerImpl (The Controler)
    # Default value is 0 which means port is randomly assigned
    # You may need to open Firewall port on the Controller machine
    client.rmi.localport=7000

    # To use a specific port for the JMeter server engine, define
    # the following property before starting the server:
    server.rmi.localport=8013

    ~I just used the port you used but actually I don’t have the idea if what it is & what would be their effect on my testing.
    The testing I want to do here is to do stress testing on this: 202.90.151.161:8013/index.xqy (deployed on our centos server) using my locally installed jmeter. So I’m not sure if it is right for me to use your steps on testing.

    -Salough

    • Hey Salough,
      Are you sure the firewalls allow you to connect on these ports?

      On top of that, what is it you try to achieve?
      You have a local install (on your laptop) of JMeter that you want to use to stress that index page on a server?
      For that you do not need a remote setup, you can just make a simple JMX scrpt that talks directly from your PC to the server. Or am I misunderstanding your question?

      cheers!
      mrtn

  8. One master and Two slave machines. Added IP addresses to properties file’s and able to connect to two slaves from Master on other applications.

    For one particular application , When i click RemoteStartAll/or any single slave machine , all i can see is the below info in logs , but no traffic is generated . Inside the slave machine , on the server cmd prompt , i see test started and finished at the same time :
    2016/03/16 09:15:07 INFO – jmeter.engine.DistributedRunner: Configuring remote engine: 10.254.72.202
    2016/03/16 09:15:07 INFO – jmeter.engine.DistributedRunner: Configuring remote engine: 10.254.72.201
    2016/03/16 09:15:07 INFO – jmeter.engine.DistributedRunner: Starting remote engines
    2016/03/16 09:15:07 INFO – jmeter.engine.DistributedRunner: Starting the test @ Wed Mar 16 09:15:07 PDT 2016 (1458144907121)
    2016/03/16 09:15:07 INFO – jmeter.engine.ClientJMeterEngine: running clientengine run method
    2016/03/16 09:15:07 INFO – jmeter.samplers.BatchSampleSender: Using batching (client settings) for this run. Thresholds: num=100, time=60000
    2016/03/16 09:15:07 INFO – jmeter.samplers.DataStrippingSampleSender: Using DataStrippingSampleSender for this run
    2016/03/16 09:15:07 INFO – jmeter.engine.ClientJMeterEngine: sent test to 10.254.72.202 basedir=’.’
    2016/03/16 09:15:07 INFO – jmeter.engine.ClientJMeterEngine: Sending properties {}
    2016/03/16 09:15:07 INFO – jmeter.engine.ClientJMeterEngine: sent run command to 10.254.72.202
    2016/03/16 09:15:07 INFO – jmeter.engine.ClientJMeterEngine: running clientengine run method
    2016/03/16 09:15:07 INFO – jmeter.samplers.BatchSampleSender: Using batching (client settings) for this run. Thresholds: num=100, time=60000
    2016/03/16 09:15:07 INFO – jmeter.samplers.DataStrippingSampleSender: Using DataStrippingSampleSender for this run
    2016/03/16 09:15:07 INFO – jmeter.gui.util.JMeterMenuBar: setRunning(true,10.254.72.202)
    2016/03/16 09:15:07 INFO – jmeter.gui.util.JMeterMenuBar: setRunning(false,10.254.72.202)
    2016/03/16 09:15:07 INFO – jmeter.engine.ClientJMeterEngine: sent test to 10.254.72.201 basedir=’.’
    2016/03/16 09:15:07 INFO – jmeter.engine.ClientJMeterEngine: Sending properties {}
    2016/03/16 09:15:07 INFO – jmeter.engine.ClientJMeterEngine: sent run command to 10.254.72.201
    2016/03/16 09:15:07 INFO – jmeter.engine.DistributedRunner: Remote engines have been started
    2016/03/16 09:15:07 INFO – jmeter.gui.util.JMeterMenuBar: setRunning(true,10.254.72.201)
    2016/03/16 09:15:07 INFO – jmeter.gui.util.JMeterMenuBar: setRunning(false,10.254.72.201)

  9. Hi Martijn,
    I have a question about report when running the distribute.
    I am performing distributed testing using JMeter and getting collective result of all slaves on Master JMeter GUI.
    Please let me know if it is possible to capture separate reports for each slave? or how do I know which request is belonging to slave 1 or slave2?

    Thanks,
    Vinh

  10. One way I have found to get around the issue of feeding results back to the master that are less intrusive than tunnelling, or port forwarding as you have documented in this guide, is to have each slave and master machines with a valid entry in /etc/hosts. Rmi seems to use host names if local host is not available and a simple entry in the slave hosts file allows the slave to return data back to the master unhindered.

  11. Hi Martin,
    I am getting error unmarshalling return … (no security manager: RMI class loader disabled) when I try to run test from master jmeter to remote jmeter server instance running on ubuntu 16.04. My client is running on Windows 10. I have followed your steps and did SSH tunneling using PuTTY. Could you please tell me what configuration files need to change values to fix this error?
    Thanks
    Sam

    • Hey,
      an unmarshalling error I haven’t encountered unfortunately. But then again, I am also not running any Windows, I run from linux to linux.
      On which side do you get an unmarshalling error? Server or client side?

  12. HI Martijn, great blog.

    My configuratin consists of 1 controller and 5 slaves. (all windows) I have set up each server and i can send the load to each slave or to All using the GUI. There is no issue there. The problem is that in the GUI (Active Threads over time) i only see the threads of 1 out of 5 slaves, i.e.: My test plan have 20 threads and i am sending to all 5 slaves, so i should see 125 running, bu i only can see 20. When the test is completed i collect the data and i generate a report and i only see 20 threads. I checked the log of each slave and the controller and i don’t see any error or connection timeout. Have you experience this before?

    • Hi Juan,
      I have had it as well in the past and I believe I made some changes to the way the logs get sent back to the master node.
      It has been a while since I did that so I am not sure anymore what I changed. I do generally now set the logging to batch-wise sending.

  13. Your article is very informative and helpful. I am facing one issue, I have 1 master and two slaves setup (AWS EC2 instances using jmeter docker images). I have placed my csv files under the slave machines jmeter bin folder and given the same path in my .jmx file however when run non gui command from master machine, csv files are not read. It shows . my jmeter slave machine container pwd: /jmeter/apache-jmeter-2.13/bin/usernames.csv. Please suggest

  14. Hi Martin,

    I followed all of your instructions, but my distributed test does not work. I try hard to do this distributed test, but something is wrong and I can not detect what it is. My slaves are on my own subnet, so I do not need the ssh steps.

    I’m running the jmeter 3.2 (last stable version) with all plug-ins installed. I allowed the ports cited in the article at the firewall.
    This configurations are the same for the client (master) and the servers (2 slaves).
    The OS of all machines are a Linux Ubuntu 16.04 LTS.

    To run the jmeter-server i have needed to use the following comand:
    ./jmeter-server -Djava.rmi.server.hostname=

    I tried a telnet connection from the client to server with success by the port 60000. At the client I used this command:
    telnet 192.168.1.68 60000

    But when I have started the remote test by the GUI of jMeter it didn’t work, it wait and wait until pop-up the message of timeout:
    “Connection refused to host: 192.168.1.68; nested exception is: java.net.ConnectException: Connection timed out”

    Can you help me in any way, please? I don’t know more what to do 😦

    • Hey Thiago,
      it sounds as if you do not have a tunnel up and running from your master to your slaves. If the connection gives a timeout it generally is the result of a problem with your tunneling/reverse tunnel.
      Alternatively it might be that you are blocked by the firewall running on 192.168.1.168

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

w

Connecting to %s