In my previous post I discussed why you might want to run your own servers for load & performance testing. In this article I will elaborate a bit on how to setup your machines. The Apache Jmeter pages of course have an explanation on how to setup remote tests. What I have heard from colleagues and what I experienced myself, is that that explanation is not always as clear and concise as may be preferred. This is my attempt at giving a readable explanation on how to setup JMeter with remote instances.
Some basic terminology
I will use the terms Local and Remote to identify the different sides of the configuration needed to get things working.
- Local is to be read as your workstation, e.g. the machine you use to build your JMeter scripts.
- Remote should be considered any machine that will be running a (headless) JMeter instance and will help generate load on an object.
How to setup JMeter locally to work with remote machines
Configuring JMeter to work with your own (or your customers) load generating servers is relatively easy, however it is not as straightforward as one might hope and expect.
Client Configuration
Firstly we need to add the fully qualified domainnames or IP addresses of the servers to JMeter. These names, or addresses will, after starting JMeter show up in the “Run” menu of Jmeter under the “Remote start” item.
In order to do so take your favorite file-editor and open up “jmeter.properties” on your own, local machine and edit the following section:
#--------------------------------------------------------------------------- # Remote hosts and RMI configuration #--------------------------------------------------------------------------- # Remote Hosts - comma delimited remote_hosts=<INSERT YOUR OWN IP ADDRESSES OR NAMES, AS A COMMA SEPARATED LIST> client.rmi.localport=7000
The two items just edited are the settings for the remote hosts, so the IP addresses of the servers you want to use within your grid. This can become a relatively long list of addresses, since you can technically add as many as you desire. When looking at my personal setup I have several instances of the remote_hosts line, where only one is active at any given point in time. Each customer that has it’s own remote machines, will receive a separate line, which is used when working for that particular customer. This is in order for me to make sure I do not accidentally start a test from servers that I should not be using or which are simply not available at the time. The latter mistake is not as damaging as the first, however I consider it annoying because JMeter will hang for a fairly long time attempting to connect to the remote machines.
The second line defines the port on which the local JMeter instance will listen for the Remote Method Invocation or RMI connections from the servers.
Server configuration
RMI ports
In the JMeter properties file (jmeter.properties) you will have to find the following settings and adjust them to your needs. I have chosen port 7000 for the client RMI port and 60000 for the server RMI port since that was a port that was open within most of my customer firewalls and not linked to anything else. Technically these ports can be left to the defaults filled in in the standard jmeter.properties. Keeping the default value for the client RMI port however will often lead to unpredictability due to firewalls being locked on loads of ports.
# Parameter that controls the RMI port used by the RemoteSampleListenerImpl (The Controler) # Default value is 0 which means port is randomly assigned # You may need to open Firewall port on the Controller machine client.rmi.localport=7000 # To use a specific port for the JMeter server engine, define # the following property before starting the server: server.rmi.localport=60000
Sample senders
Having set the correct ports to ensure all data gets sent back to the local instance for reporting in your Listeners is step one. The next step is ensuring the data is sent back to your local instance in a way that works best for you. The settings for this can be found in jmeter.properties as well, the part you are looking for is the “Remote batching configuration”
#--------------------------------------------------------------------------- # Remote batching configuration #---------------------------------------------------------------------------
This part of the configuration may be a bit more sensitive and difficult to get right. There are a lot of different settings to play with depending on the mode you choose. My preferred setting is this:
mode=Standard
This mode sends samples synchronously as soon as they are generated, thus giving you a constant update on the progress of the test.
In order to save bandwidth though, it may also be preferred to store all samples in memory until the end of the run. I prefer to not do this, since this can lead to loss of data is the JMeter instance crashes due to memory issues, but if you want to preserve bandwidth it is a good option as well.
mode=DiskStore
Connect the dots: Portforwarding
In order to easily connect with my remote servers and have them ready for testing I have created a simple set of shell scripts. They look something like this:
ssh -R 7000:127.0.0.1:7000 USER@SERVER.NAME.COM <<EOF cd apache-jmeter-2.11/bin/ ./jmeter-server EOF
So, what is hapening here?
ssh -R 7000:127.0.0.1:7000 USER@SERVER.NAME.COM <<EOF
This command will start an SSH reverse tunnel, which connects the remote port 7000 to the local port 7000. This effectively makes it possible for the server to send data back directly to your local JMeter instance, thus ensuring you can see the results of your tests on your own local machine while the test is running.
Adding the <<EOF to ensure the script, once logged in will keep executing commands until it reaches End Of File, where the script exits.
cd apache-jmeter-2.11/bin/
Once logged in to the remote machine, change directory to where the JMeter executable is installed. In this particular example I assume the JMeter instance used is not a default installed binary but a custom version. If you use a standard JMeter install on a default Linux distribution (i.e. Ubuntu) this step can be omitted.
./jmeter-server EOF
Start the JMeter Server instance and signal to the script that the end of file has been reached and no more commands need to be sent to the remote server. Instead of executing “jmeter-server” it is also possible to start the server with the basic JMeter startup script by adding the -s flag, thus resulting in the following command:
jmeter -s EOF
You can add these steps for all servers in separate shell scripts or of course make it one big setup script that by default will start all Jmeter-servers within your grid.
Now that the servers have been started it is time to run a test to see if this actually worked.
Starting the test
Starting a test with the remote machines is relatively easy once all of the above has been done. In JMeter hit the Run menu and select either the machine you would like to use or select the Remote Start All to just start all machines.
A very important thing to keep in mind in running tests on a set of remote machines is the following (as shown on the Jmeter Usermanual page as well):
Note: The same test plan is run by all the servers. JMeter does not distribute the load between servers, each runs the full test plan.
In other words, if you set your testplan to have 500 threads and you run this test on 4 servers you will effectively start not 500 threads spread out over those servers, you will start 500 threads per server thus resulting in 4000 threads.
If you do not want to start the test from the GUI there is also the possibility of starting remote tests from commandline. You can choose to either start on all the servers you have configured:
jmeter -n -t script.jmx -r
Or you can be picky and specify the machines you want to start the tests on:
jmeter -n -t script.jmx -R server1,server2
In my tests I quite often use CSV files to feed data to my tests, for example to feed usersnames and passwords, search terms etc. In order for these CSV files to be picked up on the server side as well, they need to be uploaded to the server. Where JMeter will send the testscript to the server, it does not send any data-files to the servers.
49 thoughts on “JMeter and remote servers – a tutorial”
Leave a Reply
This site uses Akismet to reduce spam. Learn how your comment data is processed.
Technical Software Testing 
Hi,
mode=Standard send all html content (for every url) from server to client each time a http reponse is receved => good to debug but no heavy load test
Since JMeter 2.9 the defaut mode is mode=strippedbatch, send non html content but another infos like size, status, elasped time … and wait for n secondes or wait for n requests before send stripped reponse to the client. => good for heavy load test.
I can’t preconized standard mode.
Regards.
Vincent D.
Hey Vincent,
indeed logging levels is a very important one to keep in mind when setting it all up yourself.
Might even be worth a separate post 🙂
JMeter’s GUI lead me to believe I could only remotely run “One” or “All” machines but your command
\jmeter -n -t \script.jmx -R server1,server2
does exactly what needed. Thanks a lot
hey Stéphan, glad to have been of help! 🙂
Running either one or all is indeed not always what you would want, I found this out by simple trial and error though.
Hi Martijn, Thanks for the information. Is it JMeter distribution testing could we able to simulate (with four agents)75,125,150,200 user’s on each machine?
the amount of users you give in the JMX is the amount you generate per engine. So that should be no problem I would say.
If you want to build up the load like that you could use the Jmeter Plugins Ultimage Threatgroup, that gives you a lot more freedome to play with one scenario and running it with different loads without adjusting the JMX every time.
Hi, I set mode to Standard on client and server node, but get nothing back. The Jmeter looks like just send all the requests out and do not wait for response before it returns. I tried to run the same test plan on the server, using local mode. It worked well. So, I have no idea what did I do wrong. Could you plz help me with that?
Hey Wesley,
Do you have the reverse port tunneling started and did you verify that is indeed working as expected?
If you are not sure your tests are running properly on the server when started remotely you can always SSH into the server and tail your jmeter logs.
What you describe sounds like the reverse tunnel not sending things back properly.
Thanks for reply. I can grep logs from the application server I test after Jmeter -R returned. So I’m sure Jmeter have sent all the request in a very short time. Perhaps I haven’t described my problem accurately. I did get summary back as follow. But what I need is the response from the application server as I get when I run Jmeter in local mode.
$ ./jmeter -n -t /home/cepm/JMeter/nodes/lt/wesley/apache-jmeter-2.13/test/EFPerformanceTest.jmx -R icam-dev-app3
Creating summariser
Created the tree successfully using /home/cepm/JMeter/nodes/lt/wesley/apache-jmeter-2.13/test/EFPerformanceTest.jmx
Configuring remote engine: icam-dev-app3
Starting remote engines
Starting the test @ Tue Sep 22 12:01:47 CST 2015 (1442894507999)
Remote engines have been started
Waiting for possible shutdown message on port 4445
summary = 80 in 7.2s = 11.1/s Avg: 42 Min: 18 Max: 115 Err: 0 (0.00%)
Tidying up remote @ Tue Sep 22 12:01:56 CST 2015 (1442894516211)
… end of run
Hey Wesley,
Might be easier to help you through either Skype (martijndevrieze or Google Hangouts (martijndevrieze at qa-rocks.com) in a chat?
I think I understand what you are looking for but not 100% sure.
I appreciate for that. Unfortunately, it is not a convenient time for a chat at the moment. Shall we communicate via email? My email address is yifaxu@cisco.com. I’m looking forward to your help.
Hey Wesley, mail will work as well 🙂
This how I configured my environments:
ON LOCAL in jmeter.properties:
#—————————————————————————
# Remote hosts and RMI configuration
#—————————————————————————
# Remote Hosts – comma delimited
remote_hosts=127.0.0.1:55501
client.rmi.localport=55512
mode=Batch
#remote_hosts=localhost:1099,localhost:2010
ON REMOTE SERVER:
#—————————————————————————
# Remote hosts and RMI configuration
#—————————————————————————
# Remote Hosts – comma delimited
server_port=55501
server.rmi.localhostname=127.0.0.1
server.rmi.localport=55511
AND CONNECT FROM LOCAL PC TO REMOTE SERVER:
ssh -L 55501:127.0.0.1:55501 -L 55511:127.0.0.1:55511 -R 55512:127.0.0.1:55512 username@host “/path/to/yours/jmeter-server -Djava.rmi.server.hostname=127.0.0.1”
You can do as I have mentioned above if you have done Ssh key auth between your machine and remote server, so remote server won’t ask you for password and will start the Jmeter-server.
Very nice blog, Sir. Thanks for sharing your knowledge on this.
I have some questions:
1. You had said ‘In my tests I quite often use CSV files to feed data to my tests, for example to feed usersnames and passwords, search terms etc.’. How do you manage the data used by server1 not used by server2? How do you handle server specific data when the same jmeter test plan is run by all the servers?
2. jmeter -n -t script.jmx -r -> This is great to start the test on all the remote servers. If i need to stop the test abruptly, If i press the ctrl+c, only the client instance is stopped in non-gui mode. Remote servers are still execute the test. What is the non gui equivalent for ‘Remote Stop All’?
Thanks Sir.
Some short answers:
1. I generally split the CSV files and divide them, if they are not allowed to crossover between servers. Most of the time however, it doesn’t damage the tests if I use the same user twice, as long as not at the same time.
2. I have a simple kill script to stop the servers if that needs to happen instantly. Not found a nice graceful way to shutdown fast.
Thank you sir. The ‘simple kill script’ – Is that something you can share with us?
Hi,
To manage the dataset (.cvs files) with multi jmeter slaves (jmeter server), a good solution is to use the HTTP Simple Table Server
http://jmeter-plugins.org/wiki/HttpSimpleTableServer/
Regards.
Vincent D.
Hey Vincent, that is an excellent pluging I haven’t used yet before. Rest assured, I am going to try to use it asap in the near future!
Hi, I am actually running this, but i get errors saying Waiting for possible shutdown message on port 4445. What can i do for this?
hey Vishal, can you be a bit more specific in what errors and what you are running?
The waiting for possible shutdown message sounds to me like a message, not an error.
Hi Martijn, i am running the Jmeter non-gui call from the master, and waiting for the response from the slave. After the Test has run on the slave, on the master i get a msg stating “Waiting for possible shutdown message on port 4445”. and the execution just stays there for hours.
Vishal,
This week I will be setting up a new server again for JMeter remote testing and I will see if I get what you have. I have not seen this issue before.
You do get your data back on the client side, right? Seems like the listener is not where you expect it to be.
Hi there Martijn,
I followed your steps on testing remote server yet I’m getting some error when I tried to do it with ‘Remote Start’. It says ‘Connections refused to host’. I don’t know where I did I missed.
Btw, FYI, I have these details with my server which I put in jmeter.properties:
# Parameter that controls the RMI port used by the RemoteSampleListenerImpl (The Controler)
# Default value is 0 which means port is randomly assigned
# You may need to open Firewall port on the Controller machine
client.rmi.localport=7000
# To use a specific port for the JMeter server engine, define
# the following property before starting the server:
server.rmi.localport=8013
~I just used the port you used but actually I don’t have the idea if what it is & what would be their effect on my testing.
The testing I want to do here is to do stress testing on this: 202.90.151.161:8013/index.xqy (deployed on our centos server) using my locally installed jmeter. So I’m not sure if it is right for me to use your steps on testing.
-Salough
Hey Salough,
Are you sure the firewalls allow you to connect on these ports?
On top of that, what is it you try to achieve?
You have a local install (on your laptop) of JMeter that you want to use to stress that index page on a server?
For that you do not need a remote setup, you can just make a simple JMX scrpt that talks directly from your PC to the server. Or am I misunderstanding your question?
cheers!
mrtn
One master and Two slave machines. Added IP addresses to properties file’s and able to connect to two slaves from Master on other applications.
For one particular application , When i click RemoteStartAll/or any single slave machine , all i can see is the below info in logs , but no traffic is generated . Inside the slave machine , on the server cmd prompt , i see test started and finished at the same time :
2016/03/16 09:15:07 INFO – jmeter.engine.DistributedRunner: Configuring remote engine: 10.254.72.202
2016/03/16 09:15:07 INFO – jmeter.engine.DistributedRunner: Configuring remote engine: 10.254.72.201
2016/03/16 09:15:07 INFO – jmeter.engine.DistributedRunner: Starting remote engines
2016/03/16 09:15:07 INFO – jmeter.engine.DistributedRunner: Starting the test @ Wed Mar 16 09:15:07 PDT 2016 (1458144907121)
2016/03/16 09:15:07 INFO – jmeter.engine.ClientJMeterEngine: running clientengine run method
2016/03/16 09:15:07 INFO – jmeter.samplers.BatchSampleSender: Using batching (client settings) for this run. Thresholds: num=100, time=60000
2016/03/16 09:15:07 INFO – jmeter.samplers.DataStrippingSampleSender: Using DataStrippingSampleSender for this run
2016/03/16 09:15:07 INFO – jmeter.engine.ClientJMeterEngine: sent test to 10.254.72.202 basedir=’.’
2016/03/16 09:15:07 INFO – jmeter.engine.ClientJMeterEngine: Sending properties {}
2016/03/16 09:15:07 INFO – jmeter.engine.ClientJMeterEngine: sent run command to 10.254.72.202
2016/03/16 09:15:07 INFO – jmeter.engine.ClientJMeterEngine: running clientengine run method
2016/03/16 09:15:07 INFO – jmeter.samplers.BatchSampleSender: Using batching (client settings) for this run. Thresholds: num=100, time=60000
2016/03/16 09:15:07 INFO – jmeter.samplers.DataStrippingSampleSender: Using DataStrippingSampleSender for this run
2016/03/16 09:15:07 INFO – jmeter.gui.util.JMeterMenuBar: setRunning(true,10.254.72.202)
2016/03/16 09:15:07 INFO – jmeter.gui.util.JMeterMenuBar: setRunning(false,10.254.72.202)
2016/03/16 09:15:07 INFO – jmeter.engine.ClientJMeterEngine: sent test to 10.254.72.201 basedir=’.’
2016/03/16 09:15:07 INFO – jmeter.engine.ClientJMeterEngine: Sending properties {}
2016/03/16 09:15:07 INFO – jmeter.engine.ClientJMeterEngine: sent run command to 10.254.72.201
2016/03/16 09:15:07 INFO – jmeter.engine.DistributedRunner: Remote engines have been started
2016/03/16 09:15:07 INFO – jmeter.gui.util.JMeterMenuBar: setRunning(true,10.254.72.201)
2016/03/16 09:15:07 INFO – jmeter.gui.util.JMeterMenuBar: setRunning(false,10.254.72.201)
Hi Martijn,
I have a question about report when running the distribute.
I am performing distributed testing using JMeter and getting collective result of all slaves on Master JMeter GUI.
Please let me know if it is possible to capture separate reports for each slave? or how do I know which request is belonging to slave 1 or slave2?
Thanks,
Vinh
Vinh, it is of course also an option to just pipe the logging per slave on the OS of the slave and then grab that later on.
I generally do that as a backup method for when errors happen for example.
I use the variable ${__machinename} at the beginning of the thread pool name. This is then logged against each log entry.
That actually is a very nice suggestion! Going to take it with me next time I am running from my own machines!
Thanks for the tip!
One way I have found to get around the issue of feeding results back to the master that are less intrusive than tunnelling, or port forwarding as you have documented in this guide, is to have each slave and master machines with a valid entry in /etc/hosts. Rmi seems to use host names if local host is not available and a simple entry in the slave hosts file allows the slave to return data back to the master unhindered.
Hi Martin
when i try to create ssh tunnel and give password of server I am getting an error “segmentation fault”. Need help! Thanks
Hi Martin,
I am getting error unmarshalling return … (no security manager: RMI class loader disabled) when I try to run test from master jmeter to remote jmeter server instance running on ubuntu 16.04. My client is running on Windows 10. I have followed your steps and did SSH tunneling using PuTTY. Could you please tell me what configuration files need to change values to fix this error?
Thanks
Sam
Hey,
an unmarshalling error I haven’t encountered unfortunately. But then again, I am also not running any Windows, I run from linux to linux.
On which side do you get an unmarshalling error? Server or client side?
HI Martijn, great blog.
My configuratin consists of 1 controller and 5 slaves. (all windows) I have set up each server and i can send the load to each slave or to All using the GUI. There is no issue there. The problem is that in the GUI (Active Threads over time) i only see the threads of 1 out of 5 slaves, i.e.: My test plan have 20 threads and i am sending to all 5 slaves, so i should see 125 running, bu i only can see 20. When the test is completed i collect the data and i generate a report and i only see 20 threads. I checked the log of each slave and the controller and i don’t see any error or connection timeout. Have you experience this before?
Hi Juan,
I have had it as well in the past and I believe I made some changes to the way the logs get sent back to the master node.
It has been a while since I did that so I am not sure anymore what I changed. I do generally now set the logging to batch-wise sending.
Very nice article Sir. I also did something similar like this using docker. For sharing the data in distributed mode, I use ReDis which is really cool. Please check here – http://www.testautomationguru.com/jmeter-make-data-sharing-easy-in-distributed-mode-using-redis/
Your article is very informative and helpful. I am facing one issue, I have 1 master and two slaves setup (AWS EC2 instances using jmeter docker images). I have placed my csv files under the slave machines jmeter bin folder and given the same path in my .jmx file however when run non gui command from master machine, csv files are not read. It shows . my jmeter slave machine container pwd: /jmeter/apache-jmeter-2.13/bin/usernames.csv. Please suggest
In your script, do you also refer to the same path (e.g. ./ ) ?
can i make one pc as jmeter client and jmeter server instance at the same time
Hey, theoretically that is possible. I’m just unclear on what you hope to achieve by doing that.
Hi Martin,
I followed all of your instructions, but my distributed test does not work. I try hard to do this distributed test, but something is wrong and I can not detect what it is. My slaves are on my own subnet, so I do not need the ssh steps.
I’m running the jmeter 3.2 (last stable version) with all plug-ins installed. I allowed the ports cited in the article at the firewall.
This configurations are the same for the client (master) and the servers (2 slaves).
The OS of all machines are a Linux Ubuntu 16.04 LTS.
To run the jmeter-server i have needed to use the following comand:
./jmeter-server -Djava.rmi.server.hostname=
I tried a telnet connection from the client to server with success by the port 60000. At the client I used this command:
telnet 192.168.1.68 60000
But when I have started the remote test by the GUI of jMeter it didn’t work, it wait and wait until pop-up the message of timeout:
“Connection refused to host: 192.168.1.68; nested exception is: java.net.ConnectException: Connection timed out”
Can you help me in any way, please? I don’t know more what to do 😦
Hey Thiago,
it sounds as if you do not have a tunnel up and running from your master to your slaves. If the connection gives a timeout it generally is the result of a problem with your tunneling/reverse tunnel.
Alternatively it might be that you are blocked by the firewall running on 192.168.1.168
When I run master machine getting error as Error in rconfigure() method java.rmi.connectException:Connection refused to host:192.168.56.1; nested exception is: java.net.connectException: connection timeout: connect
The error seems self explantory, connection refused.
So check firewalls, IP Tables etc.
Hi,
There is a web application and I want to check the performance of the server using jmeter.
Since the we cannot test the live server,make the snapsoot of that server using AWS.
Can you please tell how to configure settings in local and cloud server in that kind of situations?
As I do not fully understand your question I cannot answer it. I hope you managed to resolve it though! If not, please DM me.
Hello,
Nice article on jmeter. I am new to jmeter and trying to understand the concept of jmeter. i have one question about the below command to run jmeter on remote server on non-gui mode. where the script exists mentioned in the below command? i mean the script is in client machine or on the server machine?
jmeter -n -t script.jmx -R server1,server2
JMeter -n -t …
That is just running jmeter manually in non-gui mode, not that you run this on a server perse. That means that the path to the jmx file should be a hard path, e.g. your jmx should be on the machine you run jmeter from.